I skipped straight to a black metal sword from an iron sword.
Didnt see the small increase in physical as worth it on the silver sword.

Plus since I have a flax and barley farm im passively stockpiling black metal ore.