one, you can only have 16 capital ships
You only have sixteen. That would be a travesty if 16 wasn't already more than enough.
Yes, it in theory does prevent massing of capital ships, but in practice that's more than you would ever conceivably need.
but in the battle you described, i see no reason why the cap ship shouldnt win. one battleship should beat two frigates, thats the way war works.
My example was a simplified case designed to convey the basic idea. Obviously a capital ship can and does beat two smaller cruisers, but cost-wise it's equivalent 5-10 (depending on what you're comparing it to). If you want to use actual stats from in the game:
Kodiak heavy cruisers: 1650 shield/health each, 18 dps each
hypothetical super-Kodiak (equal to five kodiaks): 8250 shield/health, 90 dps
level 1 Marza (costs about as much as 5 kodiaks): 4025 shield/health, 55 dps
As we can see, this hypothetical super-Kodiak (something I hope I've already demonstrated is too strong for that cost level) is roughly twice as powerful as a level 1 Marza dreadnaught. The Marza does improve as it levels up, and also has loads of awesome special abilities, so at the end of the day if it were any stronger there would be some serious balance issues.