And what about ignoring shield mitigation but still dealing damage to shield?
Nope, though you might be able to give a -100% shield mitigation debuff, but that would affect all weapons. This doesn't work for armor because you can have armor values higher than 100 as well as negative armor values.
Or some how make weapon or ability damage shield and hull different way?
You can make weapons deal damage only to hull via each ship's entity file, but the damage will be fully affected by armor and mitigation. You can also specify with ability damage if you want only the hull or the shields to be damaged, but same case, armor and mitigation will be applied.