This isn't the kinda game that lets you do what you want. You either do what the devs want or mod the game.

So you can't just "build with your friends without being bothered" if you play the vanilla game. The devs have this thing about forcing people to do things their way, they don't like options or choices.

So deal with the raid and go kill the attackers. That's what the devs want you to do, so you have to.

Otherwise mod and use Valheim + and set workbench spawn radius so far out they just will never attack your base. Then you can build in peace.

That or build those god forsaken terribly ugly walls or ditches. Which I refuse to do. Why build something beautiful then destroy it with ugly perimeter. Ick.

Good luck!