Chances are you have built your base near a spawn, either a small stone grave called a Dolmen which usually has a bone pile you can loot and these can be in the Meadows - or possibly close enough to a Black Forest with a Burial Chambers or a ruined stone tower near enough the border and your base...or even worse, a swamp - either way, you are close enough that they can hear you from their hangout and will constantly come to check on you until you kill them and/or either remove their spawner/clear the dungeon, or build your base right on top of it (so that your workbench prevents them from respawning).

You will have to get used to fighting though, and also terrain manipulation and things like that, to keep yourself and your base protected - you'll know why, especially as you start to clear bosses.