Trenches work better than walls. Leave some room around the edges to account for log AOE and thrown boulders. I pretty much never need to rebuild my base, just maybe repair a few things around the edges. Don't ever put your garden close to the edge.

Also, mods. Valheim Plus can remove a lot of the tedium, or even make mobs stronger if that's your thing. (Obligatory warning: Devs are not responsible if mods mess up your game. Make backups.)

https://steamcommunity.com/sharedfiles/filedetails/?id=2906804708