Just curious, but why do you consider them annoying?

There are long breaks between them (time wise) and they are well announced so you shouldn't be surprised.

They are really just fights coming to you versus you seeking out those same fights.

And they don't pose any real risk to your base as long as you sally out and fight them in the field.

They also are some of the few fights in which AOE weapons can really shine.