A full repaired stone building or tower can take about three or four Troll blows before breeches occur. A couple more and it'll come down.

Sadly this is why Earthen rampart walls rule the defensive roost at present. That and trenches.