"The Forest Is Moving" event can have a ton of Grey Dwarves spawn if you kill them quick enough.

If it isn't the event then mining / chopping trees especially at night time can result in constant spawns of enemies wandering over to fight you.