Double recommend on the Thunderstore mod manager, it has a simple single purpose UI, a big obvious button for launching in Vanilla mode, and a system where you can generate a short hex code to paste to your friend that will synchronise all your mods and config files.

There are two kind of mods, server side and client side mods. Server side mods are usually mods that change game mechanics. Valheim plus is serverside mod. All players need to have it to be able to connect to the server.
Client side mods are mods you can use always, server does not need to have it. For example, farm grid and mass farming are client side mods.

My suggestion is to use R2modman from thunderstore. It is separate application which you can use to install or remove mods. It will not mess up your vanilla files because all the mods will be saved in separate folders. And you will have an option to run modded or vanilla game.
Also, my suggestion would be to use only one bigger/more complex mod if you don’t have experience with mods. Installing multiple complex mods could make a mess from your game. Valheim plus is good to start with.

First thing you want to do is back up your game saves in case of issues. Mods will get broken when the game is patched.

Some client side mods can be different but anything handled by the server will need to be installed by everyone on the server. Also I don't think the server will deploy the mods, people need to get their own.
You can make a guide with the mod list and where to get it and share it with people you let in.

Other than that, by compatibility do you mean crossplay stuff?

Thanks for the info.

Originally posted by GunsForBucks:

Other than that, by compatibility do you mean crossplay stuff?

No, just wondering about things like if all players need the same mod or not. Like if I installed a clock mod to see the time of day, if my friend would also need it to connect to me or not.

Hard to say if all players would "need" to have the mods installed you're running, in some instances (and I'm speaking in general here, not only for Valheim) you can join a server just fine with your mods active, just that you might not be able to use all, sometimes you get join-kicked because of a (or multiple) mods you're running (or not running, if you miss one that is installed on the server). Some mods have to be installed on the server, too.
It's always best to read the Q&A in the mods description carefully and also (in case of the Nexus) look through the posts made by others for any possible issues. Sometimes the mod author has even made a "how to" for using the mods in multiplayer.
And than there are some mods you can use regardless of if they are installed on the server or not, I believe your ingame clock is one of them, since it only affects your UI. Overall, if you don't want to install them on the server, it comes back to try and error, see if they work for you and see if your friend can connect.