For a unit with shields, the formula is as follows for damage.
Damage Applied = ((1-Shield Mitigation)*Raw Damage)/(1+0.05*Armor)
For a unit without shields, simply remove the (1-Shield Mitigation)* from the beginning while one would remove the /(1+0.05*Armor) for calculations where hull is not yet involved as shields have not yet been broken.
Let's go with the example of a unit with maxed mitigation at 60% and 20 armor. To find the amount of damage reduction, just plug in a 1 for Raw Damage.
((1-0.6)*1)/(1+.05*20) = (0.4)/2 = 0.2
That is to say, a unit with such characteristics would take only 20% of incoming damage as actual damage when dealing with hull because 60% mitigation reduces incoming damage to 40% while 20 Armor cuts damage taken to hull in half, resulting in 20%.
EDIT:
The same principle of 1/(1+Y) applies in other regards as well. If a ship were to have a buff applied to it that would increase damage by 100%, this would be extended to 1/(1+Y) where Y=1.00. 1/2=.5, therefore, damage output would actually be increased by 50%. If the buff says 20% increase in damage, that'd be 1/(1+.2)=1-1/1.2=16.7% increase to damage output.
I do not know if this same concept applies to accuracy buffs and debuffs or not.