Welcome!
My first question is if you have played a true 4X game before?
You see Sins is a 4X game at heart, it is just in a continuous time mode, what most call RTS. Just to clarify 4X stand for Explore, Expand, Exploit, Exterminate and this is what you do in Sins in that order. It plays much more like a standard turnbased 4X than a general RTS game. Yeah you can rush and do all usual RTS things if you like, it's just harder and not as effective. I'm telling you this so that you'll not try to play it as most RTS games.
I remember the same thing as you with my first combat. Try it again and do it like this: Build a capital ship factory and a capital ship, the first is free. Build a capital ship that can colonise. Build two scouts and set them on auto exploring by right clicking the icon for that on their command bar. Clear some asteroids with your cap (Short for Capital ship.), colonise and build them up. Return to your home planet with your cap, build all the frigates you can and create a fleet with them and your capital ship. Set your fleet so that all your ships jump at the same time and go attack a planet.
Now is where you can make the combat go faster. Instead of just letting them fight by themselves give them orders. A common tactic in the start is to click your fleet, have all ships attack siege frigates, colonise the planet when they're dead and then mop up.
For a more effective combat strategy target the neutral enemy ships in this order: LRM Frigates, Light Frigates, Siege Frigates, heavy Cruisers and Flak Frigates. Then go and colonise the planet.
It depends on the planet. If it doesn't have Heavy Cruisers kill the Siege frigates, colonise and then mop up in the order I wrote. Otherwise kill them all first. Well you can leave the Flak frigates for after you've colonised, they don't do much damage to your ships, they're just hard to kill.
Oh and I assume you know to build your resource mines, planet population and that stuff.
Edit: I forgot one thing. This being a 4X it's very slow in the beginning, much more than in standard RTS games. Though it will speed up as your resources start to get up to speed to the point where it's actually as fast as standard RTS or faster because you have much more to manage here than in them.