Skeletons might aggro your base if you are near a Black Forest or have killed Bonemass in which case Skeletons can then spawn in the Meadows. Skeleton Archers are probably what destroyed the hives.

Walls, moats, earthen ramparts and anti spawn buildings like workbenches can help deal with outside enemies.