Build a spike wall around your base and set up some of the stakes that stab the baddies when they attack. You can use the hoe to raise a berm around your base and/or the pick to dig a moat. You can build archer towers inside the walls and climb then to use your bow.

You can also spend less time in your base and more time exploring. The raids are checked for every so often. If you aren't inside your base at the time it checks no raid happens. (At least for the low level raids).