I'm renting a server atm and playing with 4-5 people, we have a village medium area 5 buildings and a town large area 7 building and a great hall as well as a number of outposts and camps and we get the odd lag spike with trouble opening chests and some damage to ships when sailing but for the most part no issues and the lowest my fps have gone is 40 fps using a 1070. My game setting are all set to medium.
It seems alot better than when it first came out and as long as we don't raise terrain too much it seems to be pretty smooth.

Game optimization is likely not going to happen till closer to the end of Early Access when the majority of major new content is added, I know that's frustrating for us players but that's a more efficient way to code games. If you try and do optimization with every stage, it will take way longer to finish overall and I know we're all eager for more content as it is Not the news I know y'all were hoping for I know

In the meantime, the tweaks here - https://www.reddit.com/r/valheim/comments/lizxq9/valheim_fps_fix_benchmark_picture_guide_gained_an/ - brought myself & all the folks on my server from single digit FPS in my village to 60 - 90 there, even with our 30 or so tame wolves >.>

Hope they help you out too!

Just a little update. Seems the network- lag is getting worse and worse. Now we can get quite many disconnects/ no information received etc per play session and the FPS around our base (nothing big) can dip into the 20's on very high- end hardware.

Shame really because we love the game and it would be so much more enjoyable if it ran decently and more stable.

I get this a lot while playing with my GF. We are only two on our server.
Often I will see her standing still with a running animation etc. Odd thing is it also makes me unable to do stuff like open chests.
Some of the times it will lead to her disconnecting entirely.

We have tried:
Lan/Internet
Dedicated or non- dedicated server
Vulkan/DX11
Run game as admin
Allow game in firewall