This is an odd bug I've seen pop up that is quite rare but has been in the game a while. Sadly either ignoring the fleet or building a bunch of bombers to deal with them might be your only option.

Quoting V00DOO,

reply 2

Its like 2000 fleet supply just sitting in interstellar space

Also, playing the distant stars mod so the bombers are ultra nerfed so that's a no go too.  Maybe I can catch them when they go past the planet they're in line to, since at this point they've made about 4/5 of the journey.  Almost their entire civilization has been eaten by another advent since their entire fleet is in the deep dark.

Actually that doesn't work. Even if the ships get to their destination, they will be unable to interact with anything not in the gravity well they originally left from.  

Ha, I've always wondered if this could happen when I'd see missiles going off out of the gravity well into empty space. Well if you were hard up this game, it looks like you might have a fighting chance now.