Originally posted by Zathabar:

A full repaired stone building or tower can take about three or four Troll blows before breeches occur. A couple more and it'll come down.

Sadly this is why Earthen rampart walls rule the defensive roost at present. That and trenches.

AhI The towers might be in as bad shape as the abandoned wooden buildings. That would explain it, if the walls didn't have their full HP to start with.