It might be the seed. I've been in seeds where on the very first day I found a full squad of dwarves (brute, half a dozen greys and a shaman) patrolling the meadows because there was a tiny patch of black forest around.
But yeah dwarves are so annoying with their "slap you in the back on the neck and run like a c..." strategy.

What I did in one of the worlds I made was to dig a big pit and "SPARTA!" greydwarves into it so the game doesn't spawn more in the area.