If culture is left the way it is, I'd suggest limiting cultural growth to the connections one step outside your empire.
Then if/when that site (asteroid/planet) flips to you, your culture can expand from there. Culture decay following planet loss in the periphery vs in the interior of your empire would then automatically have worse effects - makes sense, right? A planet lost inside your empire should be much harder for someone else to occupy than one in the outer limits. GalCivII had something like this, though the games and the implementation would be totally different. Culture decay after losing a planet or having cultural installations destroyed needs to be way faster - how about 10x as fast?
