Easily more than 50% of all my deaths in Valheim (including Mistlands and "work accidents") were in Swamps. So yeah I think it's by far the biggest jump in difficulty in the game.

Most of the monsters are not that hard to handle by themselves, at least if you know them, but they tend to gang up on you and some fights require some space (especially abominations) and may cause you to draw in aditional enemies. Visbility is poor, the terrain can be tricky and you're constantly wet outdoors. You can get starred monsters everywhere, and some are extremly dangerous (2-Star-Draugr will likely kill you with a single arrow).
Then there's a lot of spawners (I just started Swamp in a new playthrough and there were four spawners just in the first crypt, complete with 2-Star spawn and all), and with spawners you usually can't play it save, you have to rush in at some point.
Also, there is a lot of luck involved in how long it takes you to actually get some upgrades. Example: On my first character I cleared 3 medium sized swamps and found a whopping 2 crypts, with a total of under 40 iron, but in the world i'm currently playing the first swamp has 4+ crypts and just the first of them held well over a hundred iron. Other biomes give more consistant results.

And while it's true that it get's easier if you know what to expect and prepare accordingly, the problem is you won't know what to expect the first time unless you spoil yourself with guides, and that this doesn't change the fact that you need to prepare a lot harder for Swamp than for the other biomes.
I think a reasonable change would be to put some kind of limitation on the star levels. Mountains was basically a walk in the park after that because you can completely avoid any enemies with stars (not sure about the caves but they are not needed for the essential upgrades).