The pick axe is your friend here. Mostly troll proof except for log AOE and thrown boulders, but at most you'll have a few minor repairs. Don't put your garden on the edge. Stone walls won't save you but a trench will.

https://steamcommunity.com/sharedfiles/filedetails/?id=2906804708

Added: Don't forget to make sure there is workbench coverage over the whole thing to avoid inside spawns. You can replace the workbench with something else, like a campfire if you don't want workbenches everywhere, but it will show you where the boundaries are.