The one golum I killed was when I was inside a mining pit and the golum kept aggroing as I mined, so it stayed close. It took many many arrows to bring it down.

If you can't put the golum in a pit, then put yourself in a pit, and under a roof. A natural rock roof will break eventually, so will wood roofs, but you can repair or rebuild those. The roof will give some protection from drakes. The pit will keep the golums and wolves out. Bring frost potions and keep a fire going. The bright side is that wolves and drakes will fight the golum, and probably lose, but they will help wear it down and drop loot everywhere.

Of course, the hard part is getting that pit dug before you get killed. The golums are in the high mountains near where silver and obsidian spawns. Getting there past the wolves and drakes would be tricky. Digging a hole and getting a roof overhead before you get blasted is also tricky.

Frost potions require you to kill leeches in the swamp. The leeches themselves aren't too bad, and they spawn in any water, so you can get to them on the edge, but the swamp itself is dangerous. Drink poison resist before you go.