I agree. Instead of stone walls, my home have raised earth walls because of this. And its ugly, even with great care taken to make them nice. None of my homes can be damaged by raids, generally. Occasional hit here and there, but no more nonsense like losing everything on an inside wall from a troll log clipping through or total rebuild from losing a key support and half of it falling.