I'd suggest sticking it out. The swamps give you one of the biggest upgrades of the game. Once you grab some iron and start to make iron armor and weapons, you'll be significantly stronger