Happened to me too. Only happens when you spend a while at your base, ie like when your building something there for a while.

Kinda annoying if you leave it as they do like to smash up things, but get out there with some frozen arrows and they wont know what hit em.