When you are near him on the map, a bag icon will show up on it.
Then you'll know where he is.
In the meantime, keep searching. He can't be too far away from your starting point, but you still may need a boat.