First, I understood the anti-matter system from the start. Kinda surprised so many mention having missed it.
Second, I know Sins compresses time to near nothing, but reality would say that ships take a hell of a long time to make and jumping from one system to another would be a minor inconvenience, but not altogether a short trip. That's assuming jump times compared to construction times sync up properly. So, if we assume that people can move around and age in a ship that is phased, we can also assume that an inter-galactic jump would take a big chunk out of your life. Now I know that the Vasari are moving generations around, but it seems impractical to jump such a long distance before giving every other star in your galaxy a fair shot. The lore describes them as coming from near the galactic core. They don't say which galaxy, but that in itself implies one which we would automatically imagine, ours.
Despite all that... I still think the biggest lore issue is actually deciding what the thing attacking the Vasari is, given that it's up to the devs, not us. I did just go through the lore and it does say that they made a single error and I can't help but wonder about the original game intro where they activate something that releases a lot of energy, as if maybe an energy core or phase device. So it's likely something they unleashed, but that's still pretty broad.