>Is this seriously the only way in vanilla valheim to get metal to my base?

Correct. The devs want you to explore and travel as the Vikings did to truly appreciate the metals you've collected.

Some people use mods to allow portal use w/ metals, the rest do it as the devs want. Either play style is your choice.