They are just after you. If you manage to run out of your base just in time, they will follow you and doesnt destroy your base. But beside that, its always a good idea to build at least wooden walls around your base.