Unfortunately you're stuck with what's assigned and there's no easy way to get rid of it. Only way to get rid of it that I know of is to assign a governor to the planet, delete the thing and then remove the governor. Though I haven't checked if you can reassign an improvement.

EVEN MORE annoying is the fact that the AI always uses that sensor upgrade.

Very frustrating when you invade a colony and find a techapod hive (or similar rare resource) and you can't build an orbital harvester because the AI has already built a sensor upgrade.

You can keep leaders indefinitely after purchase without assigning them. To kill the improvement you want gone simply click on it you will get an option to destroy it. Then fire the leader. You then should be able to select a new build for the colony. Might take a turn or three however never tried it myself. 

You can just stuff those into a faction. There's no consequence for having a red level loyalty leader in a faction as far as I can tell.

There's no leader related faction events yet and the leaders have no effect outside of the modifiers for the faction. It's also a nice spot to stick future potential governors.