I think they balance between basic factories and labs and the second tier are too narrow for these improvements to be of very much use until very late in the game. I almost never upgrade any buildings until I have at least the third tier buildings.

These tier tow buildings only give you an additive percentage bonus of 5% and that is a puny increase for the time it takes to build them and time is way more precious in the first 100 turns of the game.

Here is an example...

If you have three basic factories on a planet with five million people you have a RAW output of 11 and say 100% manufacturing bonus for a total of 22 production points. Now it will take 45*3 or 135 production points (or one colony ship) to upgrade all the factories to Xeno Factories and you will gain a new bonus of 115% which translate into 1.65 production points... it will now take you 81 turns to just reach the same amount of total production from that world (if population remain the same).

But when you consider that the seven turns you need to build those upgrades is one lost colony ship you can get about 80 turns down the road is just not worth it in the start of the game in almost every case. I only see a value in upgrading improvements like this on dedicated manufacturing worlds where you can upgrade a factory in one turn and still produce ships at the same time, thus it does not matter much.

But you must also consider the increased cost in maintenance as well... a Xeno  factory is about twice as expensive to maintain in wealth so you will also need to build more wealth infrastructure to keep them alive for that very small boost in efficiency.

This same math is also true for laboratories but here time is even more important since a new technology now can be very important rather than later... not always but often it is... especially for those that give passive bonuses or new abilities you did not have before.