Dig a steep-sided ditch around the base, one that you have to sprint-jump to get out of. You can leave a small air gap for the bridge or use a single beam to walk across. This will keep out anything that walks, but not flying things. Leave extra space around the edges because trolls throw boulders. Remember to put workbenches or campfires around the base to prevent spawns inside.

Bonus: easy troll hide if you shoot them from across the ditch.

Try aggroing the locals and let them test it for you to make sure you didn't miss a spot where they can climb.