I started getting two trolls. They wander around for a bit, doing no damage. I pepper them with arrows, then they walk away into the woods and its over.
I have a wall, moat and some spikes, maybe they see no path to get near my base?