Keep in mind that the noise of mining will attract the buggers, so even if you follow this advice, you'll not be entirely rid of them. Still, one of these near your worksites (or a whole perimeter, if you're really sick of the Greys), will thin them out considerably. Mine during the day, too - there are more spawns at night.

Now, player base buildings eliminate spawns in a radius. Workbenches show their radius and are easy to make on the cheap, so they're the go-to. Throw them up with a few wood walls to enclose them and the Greys won't turn them to mulch.

There are others buildings that can do the anti-spawn job, too. Spoilers ahead.

The following structures have 20 meters of player base around them:

Artisan table
Bed
Blast furnace
Bonfire
Campfire
Charcoal kiln
Dragon bed
Fermenter
Forge
Hanging brazier
Hearth
Portal
Sconce
Smelter
Spinning wheel
Standing green-burning iron torch
Standing iron torch
Standing wood torch
Stonecutter
Ward
Windmill
Workbench (1)

Source:
(1) https://valheim.fandom.com/wiki/Player_base