Completely agree, and honestly that's my biggest gripe with the game. I kept coming out of my village to see a hole in the surrounding wall, unsure of what did it or how, and I finally assumed it's just some stealth greyling at some point, even though they can't see/hear anything interesting there.

I don't want to have to micromanage insignificant enemies, but because one little greyling sneaks up to something I've built and can keep whacking at it forever and destroy it if I failed to notice them in time.

You'd think that they'd have some self-preservation, too, not just "Hey let's beeline right for the nearest built structure(s) to smack it, not knowing who/what/how many lives there." I'd rather see, say, the greylings start lurking at the edge of town, scared to come in until they gather in numbers, if you've been too lazy to address the "increasing problem." (or til an event happens, then make sure of it by piling on.) It's sensible and less annoying tedium.
(And I even tried the workbenches etc trick trying to stop spawns. I literally had a build at one point with Great Wall of China style walls, and workbenches buried inside the thick walls at intervals... never stopped spawns.)