Mobs don't attack your walls or other structures unless they have aggro. They can aggro you, fires, workbenches etc. So maybe some of these are visible over your wall causing them to have aggro towards those and trying to reach them.

They will however walk towards you if you move / do other actions as they can hear you, unless you are sneaking. That will not cause them to attack walls, as they don't seem to aggro from sounds, they just wander in the direction of sounds.