Pretty sure they're only in the infested mines, that's where i found my ones. Though finding the mines is a pretty big pain at times. You have to check anywhere there's flat ground, be it on low ground, or on top of a cliff. Also try listening to any seeker idle sounds, if you hear some, check it out as it may lead to a mines. Make sure to explore denser mistlands too, thin and scattered ones usually don't have much space to house a building, so it's important to look out for that. Places where i did see the visvigir where in the seeker soldier rooms, and the room that goes to a drop into a pool of liquid. Though i imagine you're double checking every infested mine you come across at this point.

If worst comes to worst, go to the valheim map website, and put in your seed. And if you're wanting no "spoilers", then untick everything and make the terrain fogged, and just select the boss stones box.