Easiest way to defend against troll raids destroying your structures is to go outside so they target you instead. The first time a pair of big boys rolled up to my house I was still hesitant to fight them in close quarters, but they caught me when I was out of arrows so I had no choice. Turns out they're not as tough as they look, they just do insane damage to buildings, you can easily parry them with just a bronze buckler while waiting out the raid timer.