I never saw this as a big problem. I either transport my metals by boat, or I just built a (small) base there. It's part of the game. But then I don't mind the "chores" that Valheim give me.

It's actually the first game for me where the "chores" are not too overburdening and still remain a sense of being fulfilling.

So, for me it is not a "grind", it's simply part of the gameplay.

Edit: Majorily single player.