The answer is no, even a "meadows only island" can spawn troll and other raids when the conditions are met. However if the island is so small that there is no place to spawn anything OR you covered up the entire "red circle raid zone" with structures preventing spawning, the enemies won't actually spawn during a raid (that's what hazelrah explained on the post above).