That is unlucky. Also possibly night time.

For day spawns, dig a hole a little way out from where you are mining and punt the greys in there. That will keep them from respawning. Trolls should not be too numerous, so once you kill them they should leave you alone for awhile.

Campfires, workbenches and a few other things will suppress spawns in a small area but be careful of having three of such objects close to each other or you could trigger a raid.