There may be a dwarf spawner close by your camp. Once it's found and destroyed you should see fewer Grey eyes