Originally posted by gutlor:

So, what am I supposed to do? Run off till they completely destroy everything then come back and rebuild it with a moat? Start a new game and concentrate on all defenses first?

Greydwarfs in particular are also scared of fire. You can use a torch or nearby campfire to mess with their AI and try to bait more manageable numbers into a fight.