Any mod that makes significant changes to the server should result in a version mismatch error when trying to join. You can run different mods if they just alter client-side things and don't require installation on the server. Sometimes that gets unpredictable depending on which player is running the simulation of a particular patch of land at the time.

That said there are weird cases. There's one mod that allows use of cheat commands on someone else's server without them installing a thing, so it's all in a permanent state of Your Mileage May Vary. Some mods add whole new assets to the game and definitely need to be on every machine, others look like they're doing that but are actually just tweaking things that already exist in vanilla. A handful of mods can work purely server-side with none of the players needing to edit their game at all.