We use to have no problems even with 4 player bases in view of each other, but the last 'networking patch' caused all of us to get a FPS hit.

The game does not like when you alter terrain or over build; however, IF you have a newer rig you can alter the BOOT CONFIG in Valheim Data folder to get back huge FPS gains.

My Boot Config...
gfx-enable-gfx-jobs=1
gfx-enable-native-gfx-jobs=1
wait-for-native-debugger=0
scripting-runtime-version=latest
vr-enabled=0
hdr-display-enabled=0
gc-max-time-slice=3

CAUTION: Not for older video cards (Radeon 300 series crashes with this config.) Make sure you BACKUP your boot_config before you edit it ;)

I went from about 25fps in my base to high 50's. I hit 60 - 70 fps away from bases. Without the custom boot config I had to turn settings down and loading in took forever to render furnishings. I think the last patch scaled back some terrain caching, but I am just a gamer not a tech so who knows?

My Specs...
Ryzen 5 2600X
ASUS Radeon RX470 4GB
Corsair 32GB RAM
Crucial 500GB SSD
ASUS B450F Mainboard

Without the custom boot config I get 50% fps hit since the last network update. My friend with an AMD FX8370 & Radeon RX300 series cannot run the boot config - he locks up after about 10 minutes.

Out in the wilds I max out around 70fps and my rig is considered old now.