Loyalty gain and loss is caused by culture, which is produced by broadcast centers (or other racial equivalents). When your planet is under the effects of enemy culture, it slowly loses loyalty and when it reaches 0 the planet rebels and turns neutral.
When the planet is under your culture you get a 10% bonus to maximum allegiance (which means +10% credit/ore/crystal income). And if an enemy planet is under your culture, it will slowly lose allegiance until it flips neutral.
So, basically, make sure you've got your own broadcast centers to beat back enemy culture from your planets. 
Culture output stacks linearly, so your one broadcast center is not enough to beat back the amount of enemy culture, and that's why you still have a loyalty loss going.
