Copy and paste this to your launch options for Valheim.

-window-mode exclusive

Should disable Borderless Fullscreen. No idea why there isn't options for true fullscreen =\