So let me get this straight.. You play on massive levels with the maximum AI count (which will inevitably gang up on you) with the highest difficulty and you're getting your butt kicked by one of the most basic and least effective strategies in the game so you want it nerfed (which would actually make the AI harder).
Beyond that, the easiest thing in the game to counter is a spam. While I don't play against vicious AI's, I will say that your best bet may be to invest in light carriers and kite like your life depends on it. They have significantly more flexibility than other units as they can create fighters (which would maul an LRF spam) or bombers which, well, we all know what bomber spam does to fleets lol. Using LC spam may be the only way to survive wave after wave of enemy onslaughts.
Corvettes would also be a good choice in your situation if you had any supply available. Of course, you'd need some sort of advanced warning, so for that you need to scout.
I'm just perplexed why you want to put a cap on a single type of unit. That's what confuses me. When the enemy spams a unit, they leave themselves wide open to a counter. The fact that you died to a fleet of 100 Illums has nothing to do with Illums being OP or massed LRF being OP. It has everything to do with the fact that you were playing against an opponent that can produce effectively infinite units. Remember the old adage: "work smarter, not harder." The higher ranking AI's will only ever work harder. They will work harder than you will ever be able to. They build faster, better units than you and you're going to have to live with that. You have to be smarter than the AI to beat them.
I'm just so confused that you think a unit cap would be beneficial.. It would make the AI significantly harder because it would no longer make the stupid mistake of spamming a single unit. If you're faced with massed LRF, build mass flak, fighters, or corvettes.
The most powerful fleets contain a balance of unit types. Fleets containing a single unit type are the weakest and you seem to think that they are OP.. That's what confuses me so much.. If you're having a hard time with that, perhaps you should try a lower difficulty AI because methinks the source of your problem is that you're playing against opponents that your skill level is not high enough to defeat (no offense meant here, they do have the ability to create anything they want whenever they want after all, so not being able to take down a bunch of vicious AI's isn't abnormal).