Which means yes, if you get a troll on your wood house while you are out picking nearby berries or fishing, you can come back to a pile of wood and little left of your home.
Stay close and defend it, or make it unreachable, or go far away. Those all work. Being a bit away but not far enough = busted.

The nasty raids won't happen if you are not home. Its the stupid stray guys that break things while you are out of earshot.