Try going with your boat south of that island and i am sure you will find more swamp.

Also if there is a mountain next to swamp, they generaly dont spawn. At least i havet seen a single one in that situation.

But before you do that, run araund whole island and see if there is more swamp down there.