The AI is also pretty stupid when it comes to turrets. So if you just max out tactical slots, build 2 or 3 repair stations, then build a bajillion turrets right around where they jump in (make sure they are in range of the repair stations) they will sit there and attack the turrets and get torn to pieces before they finally retreat.
Are turrets or hangar defenses better? I've onlye ever built 3 or 4 turrets and they get crushed really easily.
Also, I've had the same problem when I had a fleet that was marginally stronger than the enemy's, but they kept fleeing. The trick I used was to get some siege frigates and utilize my Novalith Cannons (if playing as TEC) and destroy all their planets. Then I used the Akkan Battlecruiser in my fleet to colonize the planets. That gave me a quick economic boom while slowing down the enemy's economy. I just kept pumping out ships, mostly LRMs and sending them to the frontier.
I pushed forward with my fleet and destroyed everything in my way until the enemy fleet was trapped in a pair of planets. I sent Arcova scouts with sensor drones to each of the planets, found out where the enemy was and sent siege frigates to the other planet. Shortly after, I sent a bunch of LRMs, and sure enough, the enemy fleet sent half of their fleet to engage my siege frigates. I split up my now far superior fleet and sent half to each planet. The enemy fleet had nowhere to go so they had to flee to the planet I had just come from, all the while moving right through my fleet and getting decimated. After that, they reached one of my newly colonized worlds which had around 10 squadrons and a PJI and I finished them off there.