Valheim can't actually do local-network multiplayer, even with a local dedicated server. It has to connect out to the Steam servers for the SteamWorks API.

I found this out when my internet went down for a week early this year in 5ft of snow. My wife and I were snowed in, we had power, I had a dedicated server all spooled up and we were going to bash some Draugr.
Nope. No internet, no multiplayer. At all.

So the root of your problem is that at least part of the game data is having to go out to an offsite server and come back at WAN transfer speeds instead of LAN.