When I was fairly weak in the black forest, I quickly built a small base with 4 walls to keep me safe. You can see the greydwarves, but you're usually "reasonably safe" if you stay inside your box.

- If you don't see any of them around, cut down trees and expand your base, always ensuring you're protected.
- If you see the dwarves, just jump back in your box and and focus on building your existing walls, floors, structure, maybe even a roof.
- if they attack the walls, just ignore them until the health gets low and then just repair it quickly. Only takes seconds.

I realise this approach goes against the "lay down the foundation first" advice some people prefer, but if you're getting annoyed by those greydwarves, this is one way to handle it.