Two things that help:
1). As others have said, try to find a swamp next to a easier biome. Meadows preferably, but more often it will end up being a Black Forest. Set up a base at least 100 meters (preferably more) from the Swamp as stuff comes out of the Swamp at times and may see your base. Build a portal and put a wall or trench around it to protect it. Then walk to the Swamp.
2). As for the Abominations, besides what others have said, one thing to look out for is the occasional fire spouts you see in the Swamp. They are great for taking out Abominations if you can lure them there into the fire. Be aware you will need to deal with the creatures that are found by the fire, but they are not that bad once you see what they do (I don't want to give away too much here). Otherwise, if you don't see any fire spouts, just outrun the Abominations. They are relatively slow, though if you are swimming they can catch up to you.
Further along, once you get a better axe made you can take them on with that and a good shield. But the Bronze Axe will take a while, so hold off if you can until you can get better gear. Just avoid them.

Good luck!