cheese it:
dig under one of the unbreakable trees; raise the ground (you will actually be inside the tree at this point)
drop a workbench
use stagbreaker to attack through the tree and hammer him to death!
only thing that will hit you will be poison cloud...if he uses it; which he often wont at such close quarters; but if you stock up on poison potions that wont be an issue.