When I did it, there happened to be one of the old stone towers nearby. His attacks were knocking down the trees so they wouldn't protect me, but he couldn't knock down that stone tower! Any of those nearby on your map?