If the Elder location is a bad spot then just find another Elder location. They should be roughly equidistant from the center and spaced out around. Just go to the opposite side of the map and find a locator stone.

You should not be going through mountains to get to the Elder. No need for that. Your brick wall is self inflicted.