First and foremost, ensure the mods you are playing are compatible with your version of Rebellion...
Was there ever a time you played this game with mods and it worked? I don't know all your circumstances, but if you upgraded your OS to Vista then you may not have the proper drivers for your graphics card....
I bring this up because some mods that have graphics changes don't always work in the same way as vanilla...as an example, some mods cannot be used with "highest" graphics settings (you have to use "high"), though I'm not aware of either of these mods being such examples...
Permissions could be an issue...try restarting steam as an administrator instead of just the game...
I highly doubt running in a compatibility mode is going to help (it may even make things worse)...the game should definitely run on vista, the issue has to lie somewhere else...