I have no idea why people prefer controllers over a mouse and keyboard for games like this but some people do. Unfortunately a lot of folks are having controller issues.

Here are some posts/articles about controller issues that I found, they may help. There are more out there, just use your favourite search engine.

https://guides.magicgameworld.com/valheim-fix-controller-not-working-on-pc/

https://www.gamenguides.com/how-to-fix-valheim-controller-not-working-issue

https://www.reddit.com/r/SteamController/comments/m6wa9s/valheim_controller_issues/

https://www.thenerdmag.com/how-to-fix-the-valheim-camera-issues-on-pc-steam/

https://steamcommunity.com/app/892970/discussions/0/3094509155627896500/