According to https://valheim.fandom.com/wiki/Events "They sought you out" is enabled by killing Yagluth, and disabled by killing The Queen.

Additionally, the raid cannot spawn seekers or seeker brood with stars; the little yellow ones you were one-shotting can be one-shotted with a level 1 club, and the larger ones (the actual seekers) only had a health pool in between a no-star and a one-star Greydwarf Brute.

Seekers with stars are much tougher. Good luck in the Mistlands!