Go up on top of a rock and make sure you have the higher ground, then jump right on top of him and pickaxe him to death. You can rodeo him out like this for a good few seconds, allowing you do deal large amounts of damage. He will eventually throw you off. So just rinse and repeat.

However I would avoid fighting golems altogether for now. They don't drop anything of use atm. Crystals have yet to have a use in game.