If there is a biome next to the swamp that's a safe enough distance away, try building a small base with a portal to your main base there. That way you have a place to rest so you can regain your buffs, be safe from monsters, and if it gets too much to handle you can run through the portal and wait out the danger at you r main base.