From my experience, yes they will randomly just spawn in regardless of what you do. We have work benches and totems covering every bit of our area and then work benches without totems outside the walls in a ring about 3 benches wide and still randomly get spawns in our town, sometimes in buildings etc. They are easy enough to deal with though and never do any real damage.