Greydwarves don't spawn in meadows biome unless it's night. You may see graylings but they can be beaten even with your fists. Or just run away from them while gathering something more useable, they might quickly fall behind.