Keep in mind that the noise of mining will attract the buggers, so even if you follow this advice, you'll not be entirely rid of them. Still, one of these near your worksites (or a whole perimeter, if you're really sick of the Greys), will thin them out considerably. Mine during the day, too - there are more spawns at night.

Now, player base buildings eliminate spawns in a radius. Workbenches show their radius and are easy to make on the cheap, so they're the go-to. Throw them up with a few wood walls to enclose them and the Greys won't turn them to mulch.

There are others buildings that can do the anti-spawn job, too. Spoilers ahead.

The following structures have 20 meters of player base around them:

Artisan table
Bed
Blast furnace
Bonfire
Campfire
Charcoal kiln
Dragon bed
Fermenter
Forge
Hanging brazier
Hearth
Portal
Sconce
Smelter
Spinning wheel
Standing green-burning iron torch
Standing iron torch
Standing wood torch
Stonecutter
Ward
Windmill
Workbench (1)

Source:
(1) https://valheim.fandom.com/wiki/Player_base

The greydwarves are spawning because of the noise you are making when you mine. Clear cutting trees does not help. Getting rid of the purple spawners in sight of the copper vein will dramatically reduce the number of shamans and brutes spawning but not the basic greydwarves. Build a workbench at your dig site. Greydwarves will still spawn but not in your lap.

Clearing trees has some effect because the game likes to spawn creatures out of direct line of sight, so if there's not much cover left attack routes get more predictable.

Thanks for the info. I will be sure to put it to good use :)