So leave the area...either lead the greys away and pick them off or let them despawn.
The Elder won't despawn and you can go back to it after the greys are taken care of but I'd look around for a spawner first (purple glowing) as that could be the source of the greys.