Sounds like you're just unlucky.

Pick axe noise will draw greydwarves and trolls, moreso at night time so if you're trying to do it at night - don't. Sleep, get your rested buff, start again in daylight. But you can also use this as a learning experience, obviously you're in a densely populated area. Find another area to mine the copper. What I would suggest is that you scope out another spot take the time to see what's nearby. You can kill off greydwarf spawners but they do respawn. If there's a troll just look somewhere a little more further afield.