It depends on which events I've unlocked wether I build a wall or not. And almost only build stake walls to keep mobs away from my buildings, while i go out and fight them.
In meadows and BF, I don't even bother with doors, nor any sort of defences until fuling attacks.