Short answer; no, there is no way to stop them from happening unless you use a mod

If your playing the vanilla game the only thing you can do is build good defenses. Trenchs and raised Earth walls (using the hoe) are easiest and IMHO best options. Since they are terraint elements the attacking mobs can't detroy them like wood fences or stone walls.

The 'NoRaid' mod
https://valheim.thunderstore.io/package/MystroDev/NoRaid/

Also, drop a bunch of workbenches around the area. Usually If the mobs can't get a direct path to you they go after the benches first before going after your walls. Think of them as cheap ablative armor for your base :)