Most your later game slowdown/lag is all CPU based.
The game is a single core game. So thats why you see one core going full speed the rest chilling.
The game isnt super graphical intensive as it is CPU intensive.
The fighter/bomber spam is one of the worse things for the game, since it trackes each unit and Strikecraft come in squads which vary on race.
You can tweak your settings to reduces all the extra graphics which might help.
Keeping a clean and very small Empire Tree will also help.
If you get really gun ho, create a mod reducing Trade port ships/refinery ships/and Strikecraft.