I find a bow to be an effective defense against the trolls. They want you more than your stuff, especially if you are pulling the aggro by getting close or shooting them. They also happen to telegraph their attacks so it's pretty easy to jog out of range or do the roll right past them (can take a bit to get used to the timing but it's also mildly hilarious to be squirrely against them). A lot of players use trenches and/or earthen walls to keep raids away from their walls but so far I've been doing alright with stone walls that have palisades reinforcing the outside as long as I actually do face the trolls instead of hiding behind the walls.