Some cool ideas here. I'd never thought to manipulate the environment/prepare the combat area before fighting. I just now went back in for another try.

It seems I got lucky, or he bugged-out, or it's a genuine cheese tactic:

I logged back on, outside of The Elder's combat zone. I proceeded to approach the Boss area until i saw his name pop up on screen and the Boss Music initiate. At this point I took maybe another 10 paces via stealth to the point where I could see him in the distance. To start I was hidden behind a tree, (Eye Icon showing half-opened). I then popped out of cover quickly to snipe my first Fire Arrow, then popped back behind my tree. I hit him, but he did not aggro, nor did my stealth fail (Eye remained half open).

I couldn't help myself, so carried on sniping in this fashion until he was at half-health. Still no aggro. Then I thought i'd check even if it mattered i was stealthing, or if it might be I was simply still too far away for his AI to work. I now just stood in plain sight beside the same tree, no stealth, shooting away. No aggro. No challenge. No fight. Not even any Greydwarfs spawned to ambush me.

Truly a tale of two difficulty extremes. I feel like a bad casual.