the thrown rocks are a lesser damage but you still don't want them to wade out and toss rocks at your walls. Its best to be troll proofed (high enough terrain that they can't bash anything or throw a rock and hit anything other than terrain) as that will ALSO stop most other enemy like undead archers from shooting up your walls or whatever. Its also helpful if your island is near (about as far as you care to swim) other islands or mainland where things can spawn and sit there doing nothing. If there isn't anywhere else, it sometimes puts raid mobs inside or on your island no matter what you do.