VOLT and GREYFOX have the right idea.
To kill a Capitalship first, you need to figure out how many Units are you going to lose in that exchange, for example:
Your Fleet:
30 Long Range Missles - One Capital Ship
His Fleet:
30 Long Range Missles - One Capital Ship
In this situation, if you decide to hit his capital ship first, then be prepared to lose about 12 IRMs, if based on neutral grounds.
You might lose about 20 or more if you are attacking at his HW, if he has rpeiar bases surronding his capital ship, and even with all your fleet attacking his cpaital ship, it still has the better odds suruvining at a repiar bay.
If he is attacking you at your homeworld, I would aim at his IRMs not at his capital ship.
So overall KILL HIS IRMs, NOT CAPITAL SHIP.
Fleet 2
Your Fleet:
40 Long Range Missles - One Capital Ship
His Fleet:
30 Long Range Missles - One Capital Ship
In this Case, depending on the circumstnaces, AIM FOR HIS CAPITAL SHIP, then his IRMs. if based at your HW or at a neutral planet.
If at his Homeworld, then be careful, as the enemy can produce alot more firgates then you can, so if you end up killing his capital ship, you may be fighting an outnumbered fleet later on. I would usally destroy his fleet then aim for his capital ship.
Fleet 3
Your Fleet:
30 Long Range Missles - One Capital Ship
His Fleet:
40 Long Range Missles - One Capital Ship
never aim for cap, in whatever circumstances.
Fleet 4
Your Fleet:
100 Long Range Missles - three Capital Ships
His Fleet:
100 Long Range Missles - three Capital Ships
Always aim for capital ships first, becuase lsoing capital ships can demoizlize the enemy.
And remember, always let a capital ship move into the centre of the gravity well, as it will take longer for it to retreat, and you can kill it more easliy. Hope that helps.