Probably because the wall is terrain and not one of the wood walls, is my guess. Might have clipped down the slight slope. I had a similar set up for my small quarry of a copper mine and greys kept getting in. I ended up putting a wall all around the quarry and that seemed to do the trick. Quite a bit of workbenchs (kept for repair) to pull off, but those are cheap. Just be aware that they will attack said walls though.