The greydwarves are spawning because of the noise you are making when you mine. Clear cutting trees does not help. Getting rid of the purple spawners in sight of the copper vein will dramatically reduce the number of shamans and brutes spawning but not the basic greydwarves. Build a workbench at your dig site. Greydwarves will still spawn but not in your lap.