Build a deep wider moat and/or build on a higher surface. Basically deeper/higher than the height of a troll.

https://steamcommunity.com/sharedfiles/filedetails/?id=2636603224

https://steamcommunity.com/sharedfiles/filedetails/?id=2636595822

This was my first 'castle'. So bare with the ugliness. It has since a generation later, looked better and the moat even deeper and wider. The moat is actually deeper than the surface of the water, and I place spiky things just under the surface of the water along the base. For my bridge, I leave a small gap, so mobs can't get into it.

https://steamcommunity.com/sharedfiles/filedetails/?id=2643994173

Look at the backside of this build. Trolls walk up and just stare at me or try to hit me, but can't do any damage.