What are you using to install the mods with, etc? Once BepInEx was integrated into Vortex mod Manager from the Nexus for Valheim I stopped having that exact issue, no matter what mods I had installed.