invasions happen when there are 3+ objects in 40m radius near player that can be "invaded". there is a long list that includes beds, workbenches, fire sources and etc.
so one way to prevent it is not to have them in one place while building (it can be hard though).

also they don't happen that often. a bit less then once per hour usually.

and if you're not against mods you can install... ehm. i think it's named custom raids or better raids.. or something similar. with it you can disable all events.