Indeed its a large white sack on the map, you will find in a black forest
What I did once I found him in several seeds craft a portal to him give it a name or number so you craft another portal at your home and return to him when needed to sell or buy more fish bait etc.