There should be multiple Elder altars on your map. Each vegvisir points to the "nearest" one. You could sail and explore some other forest areas trying to locate a vegvisir that points you to a better location ... or you could use https://valheim-map.world/ to look at your map seed to find an Elder location you like.

Just bad luck on the one you found. Starting a new world is an extreme solution ... depends on how much time/effort you have invested so far.