The more instances (building pieces) you have in an area, the more the game will tax your computer and your FPS will dive. Unfortunately optimisation and Valheim don't travel in the same circles. But I've not personally experienced something as serious as what you have, just my FPS dropping to about 25-ish and occasional lag spikes. I have a fairly decent system but not crazy fast.