If you have a low spec system - laptop? the biggest problem will be file access and ram/vram. If you don't have this on an ssd, defrag your drives, it'll lower the access times of files to improve performance by a percent or two - shaders and textures are the biggest hits.

Lower the resolution you play at, this will reduce the ram/vram usage as well as the cpu/gpu requirements. The only graphics settings that matter are LOD (level of detail), anti-aliasing, shadows, and anisotropic filtering, turn those all to their lowest or off. Most other settings don't do anything performance wise.

The FPS drops when you move are because of shader and texture loading.

If you actually do have 50 mods, that definitely can add to your low fps symptoms. Like others have said:
There are fps improvement mods, Vulkan can help some users, and the baseline settings like screen resolution and other in game options being lowered.

You need to get a control test to see how much of it is your system and settings vs mods by playing without any mods. If the difference between control and modded isn't that great, your mods are fine for now.

If the difference ends up being 5fps+ consistantly(which is 25% of your low balled fps) I'd blame the mods and start looking for ones to cut as well as looking for options to increase baseline fps.

build nothing

if u can use vulkan then use it

yes, there is a mod called ValheimFPSPlus by KillerGoldFisch. It allows you to access some settings that game doesn't allow you to change. You can find it on thunderstore.