It seems to be because Trolls aren't as easy to read because their attack animations are so long and you have to get your parry just right. Compound that with multiple attack types and the fact some carry a log and it gets more complicated. Golems are like this as well but to a lesser extent.
Smaller critters for the most part are easier because they have faster animations so as soon as you see it start you can hit parry without worrying about it being too early/too late.