Do you have a screenshot or replay?  That seems very weird.  There's no ability in this game that will cause your units to attack each other en-mass.

Well, if it was against a few psy rapture battlecruiser, ship can be definitively captured by using the "domination" ability... myself, i use it for steal Hosiko from TEC enemy fleet...

By for Vasari, i have never see some capture... unless the game was with a mod?

And we would like to help you, but we are not sure what is going on ourselves. Vasari don't have the ability to turn your ships against you.