You cheat yourself out of the adventure to be had while schlepping ore to where you can use it. And the fun of planning and building a system of mining bases close to the ore. ♥♥♥♥ will happen, aka adventures. Nasty, uncomfortable things, adventures.

Putting obstacles in your way is how Valheim, or any game, generates stories. If you make an end-run, you miss the chance to be attacked by three trolls and eleventy-eight dwarves and draugrs. In the rain. And then come here and brag about it however it turned out.