Consider yourself lucky. There was a point in time with this game where massed carriers of any race REALLY ruled, and in my opinion (which is disputed by some, I guess) it was uncounterable except for massed carriers of your own. The so-called "counters" to the carrier (light frigs and flaks) were at that time useless (in my humble opinion - others here will differ). Even worse, carriers were the fastest ship around, next to scouts, so they could REALLY kite you. There have been nerfs and adjustments since then, and now things are a little different I hear.
If all he literally has is drone hosts and shield guardians, then ON PAPER your perfect counter is the skirmisher. I know you said you tried them, but are you absolutely sure you built enough of them? ON PAPER you should cream him with those, but note that you will need to build a TON of them - several per cruiser you need to take out (3 or 4 at bare MINIMUM I guess). You will need a TON, but note that you will still be spending less money to take out his army than what he spent to deploy it. On paper, that is your counter, and also research an ability for your skirmishers called "interference," and if you can, "reintegration." This is a rough number off the top of my head, but if he has 20 drone hosts and 10 shield guardians, I'd say you're talking 90 skirmishers minimum (at least back in the day). Did you deploy that many? A 1 to 1 ratio isn't going to cut it here.
Now, that's all ON PAPER. But I can tell you that what is "on paper" didn't do jack crap back in the day before carriers were nerfed and adjustments were made. But this is the first thing I'd try right now, if I were you. Just spam Skirmishers.
A possible wrench thrown in your works: If he has a Progenitor Mothership sitting with his carriers and guardians, or even worse - two of them - you are screwed, and this simple strategy just became a hell of a lot more complicated than what can be described here. Suffice to say, you will need to take down those Progens with something that drains antimatter (a Kortul perhaps?). Good luck trying.
Other notes:
You said you tried heavy cruisers. Don't use those if he has bombers in the grav well (and apparently he does), as they are the hard counter to your heavy cruisers. In other words, your heavy cruisers aren't countering jack crap in that situation, rather they are being countered themselves.
Flak used to not be effective back in the day. It might be now, but note that if it is, it will only be effective against fighters, not bombers so much. Plus, you are really only countering the strikecraft themselves with flak, not the carriers. You really want to kill the carriers themselves. If you try it, Flak should be used judiciously and specifically and with purpose - say jumping in with a ton of those first, letting them stay in the grav well for a while killing all the strikecraft (IT WILL TAKE A WHILE), and only then jumping in with a fleet to deal with all the cruisers.
I know you said you didn't want to use your own carriers against his carriers (I didn't either back in the day - I'd take a loss, just to try something different). And I know you said you tried carriers but he rebuilt strikecraft too fast. This simply means that you didn't deploy enough carriers. If you deploy an equal number of resources in carriers to his, and set them to fighers, you WILL either win or draw against him, but not lose. Note that the vasari carrier is cheaper than his, and has less strikecraft, so you will need to deploy MORE carriers than him, but not necessarily more resources in carriers than him, comprehende'?
Lastly, pick a Kortul as your opening capship, and nurse it up in levels, getting the "jam weapons" ability for it whenever you can.
Keep it simple first, though. Just spam a crapload of skirmishers (A CRAPLOAD), and buy those special abilities for them if you can. Report back here with your results.