Try running and climbing up those giant horizontal trees that are spread throughout the biome. A lot of them are high enough that only the ghosts can even attack you including the archers. Then you can snipe them all and clear the area unless you are near a spawner then you can snipe the spawner and destroy it then clear the area. Also only go into the swamp during the day at night I believe all biomes spawn MORE mobs than in the day. I run around in level 1 troll armor with only a club bow and stagbreaker, have no issues unless I act foolishly and allow the mobs to build up or get jumped by daytime ghosts I didnt see coming LOL Key is remembering that your stam doesnt recover as fast when you are wet so dont run everywhere and use up your stam cause everytime you are low on stam its gonna bite you with a mob.

Side note if you really feel its unfair you can open up console and devcommands and "raiseskill run some number" and then you are back to where you were before just an fyi if in single player. With console commands in single player you can "right the wrongs you feel" which is easier and faster and more effective than coming and dealing with the forum tbh