I have my settlement surrounded by walls and workbenches/torches/wards spread out to ensure mobs spawn outside the walls. As long as the mobs don't have line of sight on you, they won't aggro.* I've never had an issue with mobs attacking walls without first seeing me, and I have a large compound to defend.

*Edited to add that they won't aggro on walls themselves. If you have any other structures outside your walls, yeah they will go after them iirc.