Keep looking, they're out there. If you sail by a swamp outside the wet weather radius, you can still see clearly inside. Look for the green torches as you go. Watch out for swamp creatures coming out to greet you.

But, if at all possible, find a swamp where you can walk in from a safe biome. The swamp is very unkind to boats, especially in a new area that hasn't been cleared yet.

If you don't want map spoilers, but really want to know where to find a good swamp, you can post your seed and someone can look for you and point you in a direction.

The mountains aren't that much more difficult than the swamp. You can usually avoid golems and you might as well because their drops aren't necessary. An atgeir will stun wolves before they stun you, and drakes don't hit that hard and can be brought down with bows. Frost potion, or just regular fire, takes care of the cold until you get a wolf cape or chest piece. Not saying the mountain is easy, but it's not as big of a difficulty jump as the swamp. (Unless you get a hunted event, but those don't happen all the time.) Frost caves can be a little more challenging but they are optional, depending on whether you want the fenris set which has speed bonuses.