You neither need a lot of silver nor black metal. I'm pretty sure 20 stacks of silver is more than you'll ever actually use. In my opinion only iron is needed in excessive quantities - especially if you want to build something fancy.

That being said Valheim is very grindy overall. Personally, I occasionally dupe some items to ease the grind, because clearing yet another field or crypt isn't fun for me. I've heard there are mods to reduce the crafting costs as well. I'd look into those in case I'll ever go for another Valheim playthrough.