Well, being "harder" would have to mean something different than just more enemies, health or damage. At least for my more "advanced" defenses, having more enemies, health, or damage, would change nothing. They still won't get in and in some defenses, they won't even spawn.

So overall, I would have to say to keep it where it is. Unless raid or game mechanics are changed, I don't see that making raids "harder" really changes anything.