I had this problem as well. What worked for me was to go into the game's properties in Steam. Select the Controller section and in the drop down select "Disable Steam Input". This turns off Steam's meddling with controller input and lets the game handle it.
I'm using an Elite Series 2 and if I turn it on after Valheim starts it never gets detected by the game. So that solution didn't work for me.