I think I saw a dev comment that they are currently spawning more than they should and that that will be fixed. They should be the equivalent of the stone golems in terms of rarity.

Personally, I've found them left and right and got the armor really quickly. Maybe try a different swamp.