The mobs involved in the raids depend upon your progression with the game bosses. Basically, when you kill one, enemies from the next level start raiding you.
The good news are that the raids will trigger relative to your own location at their start rather than relative to your base.
They are also more interested in killing you or anything alive that you have, such as bees, boars, wolves or loxes. So, a "dead" base is not as interesting as a "living" one.
The bad news are that they can swim or wade from the direction of the sea as well in order to try to get around your walls.

Edit: And yes, the raids also depend upon what mobs you have gone against plus some only trigger in certain areas, as Flare said.