The key to raids is, as @TangoBlue says, they are after YOU.

So I have found the best option is to go out and fight them in the field away from my buildings/animals. They will only destroy your stuff if they cannot get to you; as in you are cowering behind a wall.