Greydwarves aren't a problem. Just build a few camp fires around your perimeter. Boars, Greylings, Greydwarfs and Necks fear fire and will flee from it.

When you are ready to take on the greaydwarf swarm use an atgeir. It has cleave and doesn't have the damage penalty for attacking multiple targets. Use the special attack and take a few steps backwards to recover stamina and repeat until all the greydwarves are gone.