What I suggested will fix your problem as well - the problem is that ships aren't removed from the group when they leave the fleet. If they did, the behaviour you desire will also exist.
Well, see, the problem is if any hotkey group contains the fleet leader, it will fill up with the rest of ships. So you can't really have a "cap ships" hotkey group, because it'll end up with everything that joins the fleet. You could do it after the fact, I imagine, but then you always add ships to the fleets.
I know you posted about changing the fleets into its own entity and not tied to the fleet leader, but that would be rewriting the whole concept pretty much, so I don't think that's going to happen. I don't think the engine is capable of interacting with an abstract object, which is why it has to be tied this way to the fleet leader.
So, assuming the fleet mechanic stays the same, having ships leave the control group when leaving the fleet only solves half the problem. The other half of the problem where you have your capital ship group filling with all the new ships to the fleet remains.
Hmm, I just got an interesting idea to test, will post back..
Edit: Well, tested and there is a workaround.. just assign the fleet leader to two hotkey groups. If you have it as group 1 and 2, new ships will join group 1 but NOT 2. So essentially you'll need one extra hotkey, and will end up with one for the whole fleet, but the second can have all the capitals but no extra ships.