Never.

Seriously.

You will be happy if you ONLY have to deal with Greydwarves. Seriously.

Tip: If you have your home built close to a Greydwarf nest, I would recommend either taking it out thus lessening the spawning of them, or use the nest as an endless resin resource.