To troll proof is hard, however its doable. Remember that trolls have a large reach, so the ditch needs to be set back far enough from anything valuable you want to keep. This way you can use stockade walls instead of stone walls. If you look at many viking settlements, wood stockades are not out of place. I also built a lovely building, ground shook, and it got trashed, thats when i started building moats. You can hide moats behind trees, and put lilly ponds in not too deep water to make it more attractive, and reed beds.

Oh and raids can be anywhere, mobs from the wrong biome can show up, so you can't move away from the problem.
As Intellect said, only option, disable raids, if it bothers you.