They are easy to deal with.

As you progress through the game they get tougher but they are never really more than just an annoyance. and if you build a moat around your base then you are basically untouchable by the raids