Depends on the mod I would say. You should give some examples. Graphic mods are all fine. Valheim plus and that loot mod too. As long as you guys have backups of the world nothing should happen.

The most important thing is that both of you have them. Not only the host of the world.