It depends on where I'm at in the game. I find that a stake wall works until I hit swamp and get stone. If it's a troll raid, I might have to draw them away, but the stake wall slows them down enough that I haven't had a problem with them doing much damage.

After that, I generally build a castle wall using a double layer of 4x2 stones, and have never had anything get through that. I've only ever had one full fulling raid, though, and it was in the construction phase, so I just dealt with it. I don't know how the double stone walls would fare in that situation.

I've never felt the need for moats or earthen walls.