I actually found one of the altars before finding a pointer in my previous world.
I usually struggle with Moder, takes about a dozen mountain zones to find a pointer.

You could also pull up your world seed and look at the map and find one of them.
Or have a friend look at your world seed and give you a hint of a general direction.