The only time you don't get all your mats back is from Campfires and Torches/Sconces.
If you have a main base you can portal back to you only need to carry the mats to build a portal. You can find wood/stone to make a workbench and campfire almost anywhere. If you're more nomadic I'd suggest building a cart to carry stuff in. You really don't need much... 5 Surtling cores for a Charcoal kiln OR Smelter. Everything else you'll need you can find on your journey. Maybe carry two iron ingots if you have them. Then you can build a Stone Cutter. I often find halfway decent Stone Towers I can refurbish in the Black Forest and use as a base for smelting copper/tin, and if you're lucky and find one with Swamp and Mountains nearby you can do Silver and Iron too!