That's either a raid because you killed the Elder, or you're playing on a server and someone else sent them after you. Was there a big red circle on the mini map? Killing a boss unlocks raids for the next biome up.

Digging a trench around your entire base solves most raid problems until you get to the mountains.