Trolls deal a TON of damage to stone. It's not because stone is particularly weak, but rather because stone pieces are tiny and troll attacks deal huge AoE. So they'll whack a dozen rocks at once and it'll hurt.

Wood stake fences are great for overall defense. They good endurance and are fairly cheap border protection. If enemies can't smell the player nearby they tend not to aggro, so a wide perimeter is very effective.