It might not seem like it but those greydwarfs are offering you a lovely opportunity that will last you right up to the end game.

Because once you get yourself a club or copper mace they offer you the chance to start levelling that weapon with their lovely squishy exteriors, you can also add a shield if you want to practice blocking and parrying fairly safely. Also practice dodge rolling away from attacks. Most of the stuff they drop will always be useful also.

Once you had enough of playing with them and you'd rather just get stuff done dig moats around your base or at least the important parts and put up some stake walls to stop the annoying pebble throwers hitting stuff.