You're not totally wrong. Bombers are good destroying structures and fighters are good for shooting down other fighters and bombers, but that isn't the extent of it.
Fighters are good vs other fighters and bombers and they are also good against long range frigates (TEC Javelis, Advent Illuminator, Vasari Assailant). Fighters are also good against scouts and civilian ships (constructors, trade ships, etc).
Bombers are good against structures, as I mentioned before, but they are also very strong against heavy cruisers, and are pretty good against capital ships as well. They're mediocre against frigates and support cruisers.