I got a raid centered on a swamp crypt once and only had a portal, workbench and chest. Luckily I had been paranoid in advance and surrounded the top with stake walls. Now when I do outposts I only have two of the three up at any one time. If I need the chest, I take down the workbench.

If you dig a steep-sided ditch around your whole base it will keep out anything that walks. Align yourself with the compass if you want smooth walls. You can sprint-jump out of there but they can't. Raids get tougher as you kill more bosses, and watch out for things that fly or throw things.