The raids are there to kill you, not destroy your base. Your base just happens to be between them and you. When a raid starts leave your base, or sit in your base and watch it become materials again.