As you are exploring, pay close attention to your full-screen map as a bag icon will appear in his location. As Ren posted, "He can appear in several different places, but will permanently stay in the first place you found him." In the lower left hand corner of your mini-map a bag icon will appear as an alert confirmation to Haldor's determined location.

If the first place Haldor appears on the full-screen map is an inconvenient location, do not get close enough for the bag icon to appear with the mini-map and one can keep looking for 1 of the other 9 possible locations in the Black Forest.