Originally posted by wizard_of_woz:

Until you get the Drake event, then walls and moats are useless. I find that event the hardest to deal with.

I only dealt with that once in my last full playthrough, and they barely broke through my wood roof/wall pieces and couldn't really touch the stone. I guess YMMV but as long as you have everything important roofed they can't do much.