Sounds like particle generator problems. Things like torches and Frostner that create a bunch of extra particles could be slowing you down. Not sure if Valheim specifically has a setting for particle generators, but it's a common FPS problem in video games in general.