Try to parry - block in the last moment before it hits you. That way you will stagger them and deal extra dmg while they are staggered. Keep in mind that you need some stamina to block or parry. If your stamina bar is empty you cannot block.
Also rolling in the last moment helps you evade the attack. You can roll towards them to stay close and hit more times between their swings.
Instead of axe use a spear because it deals more dmg to trolls and attacks faster.
To prevent them destroying your base, try to dig a moat around it.