The only mob mod I know, there may be others, is a mod that lets the mobs upgrade to higher levels, like GIANT skeletons with random effects, like slow or lightning... it doesn't add mobs just sort of ups the level the base mobs can go to and ups the bosses to crazy levels.
This can break the game but it does add some crazy to this pot.