cheese it:
dig under one of the unbreakable trees; raise the ground (you will actually be inside the tree at this point)
drop a workbench
use stagbreaker to attack through the tree and hammer him to death!
only thing that will hit you will be poison cloud...if he uses it; which he often wont at such close quarters; but if you stock up on poison potions that wont be an issue.
you can also go upwards; build a platform up a tree above his head height and just shoot him from the comfort of the the platform.
both methods work; the stagbreaker one might be a bit faster and less costly in materials for arrows and bow repair etc