It doesn't always work that way (which may be a bug). I've even had things spawn inside my base even though I have had full work bench coverage on my entire main base (birds and deer).
https://steamcommunity.com/sharedfiles/filedetails/?id=2860289081
https://steamcommunity.com/sharedfiles/filedetails/?id=2920176358

I've also had a troll raid spawn at my boat dock that only had a single work bench down. The boat was dismantled and inside a chest at the main base about 100 meters away. Since I didn't care about losing 20 wood. I just sprinted out of the area. No other raids can happen while one is all ready in progress and the raid timer doesn't count down if no players are in the raid area. The raid will despawn on its own after 3 game days if no players are in the area so you can't game it much but you can pretty much build in peace anywhere else for those 3 game days.