There can be astonishing numbers of spawners in nearby forest. If you cleared those out (I prefer to keep them so I can stock up on drops) it will help a ton, but nothing ever really reduces them to the 'infrequent' level of every other mob.
Once you can one-shot the brutes, you can ignore them or dispatch them as you see fit, and the problem is sort of a non issue at that point.