I'm not seeing any major drops in frames on vulkan. I do get some drops every now and then, but most of the lag is the auto-save or odd spike. World save is usually 1-2 seconds. Using full disk luks and ecryptfs so would likely be faster for most.

I have about 128 hours total, made several larger bases and have about 30 portals. That said, some of the larger bases dip into 70-90 but I mostly get 100-120FPS with vsync monitor at 120HZ.

Specs:
Mint 20.1
AMD 3900X @ 4.2GHZ
32GB 3600MHZ
nVidia RTX 2070 (460.32.03)
1920x1080 Everything on HIGH, vsync on and no DOF/Motion blur or CA
Corsair M600 PCIe 4.0 5GB/s
Cinnamon 4.8.6

Also note, that the unity fix for multithreaded jobs helps a bit. (not much in my case)

Add that to top of boot.config in your valhiem_data folder.

EDIT: Also, you could try turning down draw distance, that will likely help a bit.