I installed a mod that basically allows you to pause the game. It not only stops time but it stops graphically rendering the game and shows you a screenshot instead in order to save resources.

Despite the game being paused, task manager shows that valheim.exe is still using 30% of my CPU. * WHILE PAUSED *.

Yeah, I'd say this game is still unoptimized.