If you are close to land like a shore they will spawn at the shore and try to travel your island but if your island is tall it will be fine as long as there is no point of access that lead toward your base and they can't destroy stuff, i however have an island with such access but it's still fine because most enemies didn't cause issues so far. still when mistland is out i'm probably going to find an island in Mistland biome that is isolated enough so i can build my gothic House or Cathedral even.