Unless you built like right next to his altar you can just walk away, get some better gear and then destroy him.