If the workbench area covers all dry land and shallow water, you're good.
Walls only protect from spawns walking into your area, but on an island surrounded by deep water for a decent distance, this will not happen.

I'm more miffed about spawns deciding to swim all around my town to get into my docks area *which they cannot see*, messing up my fleet.
Definitely need to cover all spawnable surface for a good distance to prevent that.