You can usually ignore the issue of whether a mod is client-side or server-side by just making sure that everyone has the same mod setup.

The easiest way to do this is to set your mods up so that everything works the way you want on a local game. Then, once you're satisfied, go into your /[steam install]/common/Valheim directory and copy the following folders and files

BepInEx (folder)
doorstop_libs (folder)
unstripped_corlib (folder)
doorstop_config.ini
start_game_bepinex.sh
start_server_bepinex.sh
winhttp.dll

COPY these into a .zip file and upload it to a file hosting service like google drive and have your friends copy the contents into their own /[steam install]/common/Valheim directories and your will all have duplicate mod setups. Even if you've custom configured the mods, that's all included and they'll have the same config.

There are very few exceptions where this doesn't work, most of which are mods which just weren't designed to work with multiplayer. But it's only a subset of those, since even if a mod is client-side only, everyone will have exactly the same client-side mods.