My first move for that event is to leave the breakables behind and run to the other side of my moat with an iron sledge. Some kind of barn or shed for at least a few of your prized animals will keep you from having to build a gigantic structure; drakes will kill your boars too.