Yes. You need to build a castle. Make sure it has no windows, or the bats will just fly in through them and murder your boars when you get the bat raid.

Also, a perimeter wall around it so no enemy attacking the building will clip through the wall to kill them.