If your colony falls under the influence of another civilization, and you build cultural buildings and/or starbases to increase your influence to the point where your colony is once again within your sphere of influence and is no longer in danger of rebelling, the yellow exclamation point and warning tooltip never go away. In my case I built a couple of consulates, a couple of cultural starbases, upgraded them, and now have expanded my influence to the point where the nearest AI planet is now within my sphere of influence and starting to rebel, and yet the yellow exclamation point next to my planet's influence is still there, and the tooltip is still warning me to build more influence improvements.