Not every swamp has them. Look for the short rectangular buildings while sailing by, and then walk in from a meadows or black forest is at all possible. Used to be able to spot them from the green torches, but it seems that the load order had changed, so the torches aren't always visible from outside.

Smaller swamps and those along a narrow strip next to a mountain are less likely to have them.

Some swamps have a lot of them. It's completely random.

Hopefully you found turnips in one of those other swamps. Those can show up in any swamp.