I also use a controller exclusively now that I'm older and holding my left hand in WASD is incredibly painful for extended periods of play...building works fine here.

...the only real 'glitch' using my controller is that when splitting a stack in Inventory and moving to a container, I'll get my cursor passing over into the Status Menu and that's annoying. I've started to split and move to an open slot in my backpack Inv, then using R Trigger and [A] to move that slot to the open container slot...been working pretty good. Rather than splitting the stack and then using the Dpad to change active window to move the stack...that is where it bugs out[for me] and opens the Status Menu as well, sometimes going as far as engaging Friendly Fire!

I agree, Controller support needs some help/love...