Thanks for responding, I play with keyboard and mouse so have not experienced what you are describing.

I suggest going to https://valheimbugs.featureupvote.com/ and add a new bug, as you are not the first person to report it on the forums. It may be a conflict between steam and Valheim when using a controller, or if you are playing on Xbox it could be an error in the way that has been integrated. Also check your steam controller support settings.