Its not the height. If its about 2 vikings tall its more than enough.
what I do is make a circle about the size of the new game standing stones you start at, and a gap you can just barely walk through. He will spew adds into your arena but it should by this point just be a moment to clear them out. Then melee toe to toe at the door and if you need to retreat, he can't come in there. Make it so you have a place to hide out of LOS of the door. Then its just a melee battle, and fairly easy. This may be overkill now, but back when poison was hard hitting (when a GD shaman could take off half your health with one fart cloud) it was my best strategy against this guy (barring skip to mountains approach).

If you do this, its not an epic castle foundation, its a little arena. It can have gaps, it just needs to keep him out and offer a spot out of LOS where you can heal for a min. You often need a portal to go repair and refresh your stamina rested buff, it can be a long fight the first couple of times.