Build stake walls all around your land, and make sure the middle is filled with things like workbenches and campfires (they stop enemies from spawning near them). This at least makes it safe to roam your base without having to worry about a sudden raid spawning enemies a handful of feet away. If enemies do spawn, you can quickly run to eat some food if you need it, heal at the campfire, then engage them before they break any walls down.

If raids really bother you, there are mods to disable them.