This is not a solution.. this is for understanding how MP should work

in any MP game sharing the "same" world" is quite problematic.
Firstly "size of the pipe" how much data can fit between ALL computers attached (we should think of it as ping and Packet loss when the pipe is too long too slow or too busy)
Secondly "what is meant to fit down the pipe" if its Cod or MOW .the world is designed to only show certain people things in their locality, otherwise you have to update EVERY change to every player.. so its limited to bullets, bodies smoke etc,
Valheim you can Change the total look of the world piece by piece (With no regard to the amount of data that needs to fit down the pipe) so the other player can see the exact same thing... not just changing a door, but building an entire city MUST change the MP experience.. ie slow it down or crash it.

i have built 64 vs 64 maps in unreal and the available "viewing distance" is a headache.. so you end up with little rooms, hallways valleys etx to isolate players from even POSSIBLY seeing all 63 other players at the same time.. but if WE change the Whole view, leveling fields trees etc etc . then the view draw distance suffers incredibly// AND ALL that has to fit down the pipe... AS WELL as your foes spawning..

Finally.. Valhiem devs are wonderful to allow the building within worlds and ALSO have a MP experience// but the "pipe " is only so big.. you may be pushing its limits. minds greater than mine may have tools to measure your pipe and the data you are shoving through it.. might i suggest, trialling an alternate less modified world and seeing if it crashes... ?? Good Luck