Under fleet commands in the lower right corner of the screen is a button that says "auto-join". If there is a little arrow chasing itself in a circle, it's activated, right click to toggle the activation.
If you have a fleet sitting in the grav well with your factory, you might try doing the same thing with the fleet. I'm not sure here, as there is no option to build ships with auto join already turned off, but that might stop units from joining that fleet. Hope this helps.
