as a simple solution - if a raid happends - go out and fight or kite until the raid ends
after tons of gameplay i only lost 2 fences and badly placed beehive during raids by simply going out - no matter how big or small i build

keep some space between the buildings and no enemy will stop to break your home
also dont block pathways just let em flow thru

more moats and walls mean enemys will tend to break those because it blocks their movement - ye more obstacles ye higher the risk of losing beloved builds

still if you want trenches and stuff i cant help