Take them out one at a time, then retreat to safety. If the Greydwarves are too much of a hassle for you right now, maybe you should retreat from the Black Forest and make sure to upgrade your armor and food first before venturing in there again.

What are you eating and what is your armor?