Your base is collateral damage, trolls wont proactively destroy it more they hit it while trying to hit you. If you hide, they will attack certain structures (workbenches, torhces etc).

If you're going to build a wall, you need to do it right. Make sure it's all snapped together otherwise enemies will find gaps.

You then need to fill in your base area with workbenches or fires/bonfires to prevent things spawning inside the walls.