If you cleared your "home" island .. whats stopping you to explore another island? My first worlds spawn island had only one vegvisir. Elder on a totally different island.. while exploring said island. I found tons of vegvisir.

You just have bad luck with the world seed as you can see by your own since you found it in your second world pretty fast.