I've done a ton of testing on this to satisfy my own curiosity, and to try to figure out how to best use these ships and abilities. Glad this thread has come back. Look out, this post isn't a short one, but hoping some of the other Advent players can benefit from it (already have a rough time against TEC's super economy and Vasari's Dark Armada, whatever helps the fight).
To answer, reubenyap though, Malice causes all of the units hit by it to share a portion of the damage they take with all of the other units effected by it. It can be cast on any unit in an enemy fleet, and it has a nice little area of effect and it can cover a good portion of a clustered fleet. Units afflicted by Malice will appear to glow white, if you zoom in there is an effect with hands moving towards the ships, just in case you're in doubt.
The percent of the damage shared depends on the level of the Malice (Level 1 causes 8%, Level 2 causes 16%, Level 3 causes 25%). So at Level 3, if a fleet is hit by Malice, and Ship A takes 100 damage, Ships B, C, D (and so on) all will take 25 damage (however this 25 appears to also be affected by mitigation, so the damage is further reduced). The damage that the other ships take as a result of Malice sharing damage (the 25 damage in this case) isn't further shared and propagated among the fleet into infinity, but damage from other attacks for the duration of Malice will be shared similarly.
The effective result of Malice is a massive increase in the overall dps your fleet is capable of for the duration of the Malice (20 seconds). It's during this window that Cleansing Brilliance shows it's mettle.
Cleansing Brilliance is an attack that causes 250 damage a second, for 8 seconds, for a total of 2,000 damage to any given target (in a very notable, very bright beam fired from the Radiance). Ideally this attack should be timed inside the 20 second window of Malice, because it I'm very sure now it does propagate some of the damage. Cleansing Brilliance however is effected by Shield Mitigation, so doing 2,000 damage is usually only in an ideal world, and because it does not occur instantly, but instead over 8 seconds the enemy ship may start at little mitigation but end up maxing it out by the end of the attack. And since Capitals are generally the often targets (as they will not die from a Brilliance hit), they may be able to mitigate half or more of the damage they take from the attack.
So effectively the damage shared is often quite less than 25% of 2,000 (when I was testing the capital I was testing on maxed out at 68% mitigation, so a lot was lost). Messing around with this, it seems that Heavy Cruisers may often be better targets for the attack, since they have lower mitigation caps than a leveled capital ship, last I checked. Combined with the mitigation that each of the unit under the Malice may retain, it can be the case that far less than ideal 500 damage is shared.
But something I only realized because of this thread and honestly never noticed because of my targeting priorities and the chaos of battles. Cleansing Brilliance, despite being a focused beam, will damage all of the ships that get in it's way and it also has what seems to be a nice little area outside the visible beam in which ships also take damage. These ships all take the 250 damage a second of the beam as long as they stay there, and the damage these ships take count as a separate attack to Malice, resulting in 25% of thee damage they take as well being shared. Cleansing Brilliance has a rather impressive range, and aimed properly you can hit more than one ship with the beam, and each will spread the damage to the beam, and even after all the mitigation, you can be looking at sharing hundreds and hundreds of damage (a well placed beam in my test did over 1,000 damage to ships that were not even in the beam).
From all this I wish I could edit my previous post.
In practice, it seems most efficient to not aim it at a Capital Ship, but if possible a ship towards the back of a fleet (ideally a cruiser though with more than 1,500 hit points, otherwise you still risk killing it and cutting short the damage given). A Capital Ship towards the front of a fleet is the juiciest target, but it will mitigate more of the damage and the beam will pass through fewer units and inflict far less damage to the ships who are under the spell of the Malice.
This is more risky though, both Malice and Cleansing Brilliance have impressive ranges, but bringing them closer to the enemy fleet in order to hit a further back ship puts them at risk to focus fire from the entire enemy fleet, or more alarmingly, abilities that may disable one of them or their antimatter abilities, either of which done at a bad time on either of your ships will destroy the combo.
If anyone's done further testing that can elaborate on this, I would be interested in it. I never realized all of this, and really was under the impression that it was just either Cleansing Brilliance, Malice, or both not being effected by Shield Mitigation that could cause such massive hurt to an enemy fleet. Turns out I was quite wrong on that, but properly aiming and timing this ability is really is crucial, a lot more work than just focus firing a blob of Heavy Cruisers, but also great payoff.
*Note. All my testing was done in 1.03 against a Vasari AI. My fleet was present (loaded up a save for them) but my ships, including the Progenitor and the Radiance had Auto-Attack turned off and Guardians using Repulsion in order to avoid dealing damage that could taint my results. If anyone tests anything and comes up with contradictory results I would be interested to hear about them and the circumstances they occurred under. I have a bunch of saves from right before my various tests so I can retest easily (or provide pictures, which I resisted doing here) in case there is something that doesn't seem to fit.
I'm not 100% sure on the Malice damage being effected by Shield Mitigation, the numbers get so fumbled during a Cleansing Brilliance attack (the mitigation on the target goes through the roof). But all of the units under Malice seem to take less than the 25% of the damage ultimately taken by the ship subjected to Brilliance, and all of them have their mitigation levels raised higher than they were before the attack, so I assume that means that mitigation is occuring on thier end as well.