Make a smelter base in the swamp. I usually raise the earth connected between 3 or more unbreakable trees and roof it off as a base for smelting. At least some time can be saved that way so you can keep exploring crypts to get more ore while it smelts.

As as pessim said, also sleeping while it smelts helps.