I have a modded dedicated server set up in which myself and 6 other people have been playing without major issue. One of the mods we're using is Magic Overhaul. The mod causes a lot of lag for us occasionally, but we have been able to play together.

All of our modding was done manually without a mod manager such as Vortex. It is extremely important that everyone has the same game files, configs, etc.