You can raise defenses by adding a moat or using earthen walls to keep out them. However sometimes if you have a garden they will spawn inside it anyway. BTW I heard if you add the totem it will help keep attacks down. Not sure if that is true anymore with the update