Once you find a cave entrance though, you very much know it is one.

Be careful of Cultists.