you need to make sure your base is defended. greydwarfs are the least of your problems down the line believe me. there is things that CAN and will destroy your base if you dont make sure it has a defensive line around it.

either by natural obstacles (deep water for example NOT RIVERS) or by digging a moat around the base or raiseing the ground and building the base on top. there is a bunch of options includeing cheesy mudwalls but moats dont work everywhere. it depends on where your base is build.

pick your location wisely.

make sure you have open sea access in some form (river/channel or lake towards ocean atleast) otherwise ore transport will be a pain later on and on most seeds water travel is required to progress.

you can build defensive structures when you unlock them. these dmg enemies who get close by themselfs.

there is more options for defense down the line aswell. dont wanna spoil them.