Hard to say if all players would "need" to have the mods installed you're running, in some instances (and I'm speaking in general here, not only for Valheim) you can join a server just fine with your mods active, just that you might not be able to use all, sometimes you get join-kicked because of a (or multiple) mods you're running (or not running, if you miss one that is installed on the server). Some mods have to be installed on the server, too.
It's always best to read the Q&A in the mods description carefully and also (in case of the Nexus) look through the posts made by others for any possible issues. Sometimes the mod author has even made a "how to" for using the mods in multiplayer.
And than there are some mods you can use regardless of if they are installed on the server or not, I believe your ingame clock is one of them, since it only affects your UI. Overall, if you don't want to install them on the server, it comes back to try and error, see if they work for you and see if your friend can connect.