Night time spawns x10 worse than day time. I have the same issue. I have a bit of OCD and the grey dwarfs were really ♥♥♥♥♥♥♥♥ me. Best tip is to go through the forest biome near your base and kill every grey dwarf spawn you can find. (Looks like a big glowing purple/pink fire) That way the day time can be relatively quiet once you have taken them all out.
It just doesn't seem as if the devs would have play tested this very much as it is one of the worst spawning systems I've ever come across in a survival game and I've played a lot.
Trolls take a couple of days to respawn but i really don't know why we have to have a random grey dwarf scout 50m in every direction.