If you don't need it, why build it in the first place?

I think you should look at things the other way around.
It's because you build those defences you're safe. If you'd just build a single palisade or stone wall, sooner or later a troll event will smash through them. As well as through your other buildings.
Blobs will jump over them, Draugr and skeletons will attack walls until they break and so on.

But with a moat around your base and / or dirt walls behind that moat, having a spare palisade or stone walls between the moat and your buildings, the chances to have monsters entering your base has reduced a lot.
Yes, there's still bats and in worst case a Gjall, but in general it's because of the defenses the creatures that can't fly can't enter your base.
Even those that spawn only at night or from events.

Thorin :)