The hoe and the pick are your friends. You can raise an earthen berm (earthen walls are indestructible) with the hoe and dig a dry moat outside that with the pick. Leave room in the berm for a gate and build a bridge over the moat. Then all you have to do is defend the gate (and there are even tricks to get around that). Your base is safe until you get to one of the late game raids and you can limit the damage from that by building a covered barn for your livestock.