You have the basics right; trade isn't particularly useful early on, but becomes increasingly important later in the game. Exactly when you choose to start building your trade network is a question that depends a lot on your strategic position, the layout of the map, and what stage of the game you're in.
When you do get around to building trade ports, the key is to build them in a line to create a long trade route. The longer the trade route, the more money each trade port will make. When you're first building your trade ports, you want to do so in a way that quickly extends your network and gives you high returns. This factor is so important to the returns of trade ports that it's worth ignoring planets with % bonuses to trade income. While those are a nice boost, the fact is a 30% bonus to one trade port is completely eclipsed by an 8% bonus to all your trade ports. Once you've filled out your trade route, then stack up trade ports on planets with those nice bonuses.
Sometimes, you get an empire that just screams for trade ports, with a long and snake-like formation that makes creating a great trade route easy. Other times, you have an empire that has a bunch of uncolonizable "potholes" blocking potential trade routes. In this case, trade ports just aren't a good idea. Sometimes you get a location with over a dozen resource extractors within a 1 jump range, and that just screams for a refinery (or three). Refineries don't need any excessive combos to be worthwhile, just a good location with lots of extractors, so if you need to bolster your income but can't get a trade route operational, some well-placed refineries are a decent alternative.
Strategic considerations are another important factor. If you're in a long stalemate situation, then you have the opportunity to spend your cash income on trade ports and other economic investments. If you're constantly doing battle with your enemy, I'd rather put that cash towards more units or researching better types of units. As opportunity allows, trade ports are a great investment. If opportunity doesn't allow, don't try to force it.
Finally, there isn't a big difference between factions. The trade ports require a different number of labs, the TEC and Vasari get a few upgrades to increase their trade income later on, but otherwise they're the same structures with the same cost. Advent, however, lacks a conventional refinery. Instead they have an upgrade that allows them to convert their trade port into a specialty refinery that works differently from the TEC and Vasari refinery. I've crunched the numbers on this "resource focus" ability and I can simply say you should never use it. It's just a crappy ability and it's in the running for the worst economic upgrade in the game.
