Starting disclaimer: I am not a modeler. I am a scripter who can't always get art guys to do all this for me. If I misuse terms or I underestimate Max or something because I don't use it as frequently as a modeler would, it's completely unintentional.
Having icons and viewscreen pictures for units in your mod is a big step toward making it look new and polished. It took me a while to figure out the best way to do it, but now given any skinned model, I can do it in under 15 minutes. Here's a detailed guide on how to edit viewscreen pictures into the game: tips here such as how to do alpha channels also can apply to really any .dds editing that you have to do.
I assume you have Photoshop for this, as using anything else is likely going to be an enormous pain in the ass. Also, you'll need the Sins developer exe, which you can download on the Downloads page, as well as the DDS tools for Photoshop which you can Google (I'm pretty sure they're on the DirectX or Adobe's site).
Why Renders Suck
Now, let's assume you have a model skinned and ingame. You could render this model in Max, XSI, or whatever, but there's a few reasons this isn't the best solution.
1) The obvious: Max can do normal maps during rendering, but it can't really use the data map the way Sins can, because the way the data map works is proprietary. You can probably hack this by making separate self-illumination textures, but it's not really worth it to try to approximate ingame's. The worst case scenario is that nothing in your data map will be shown in the final product - no bloom, no illumination, no reflective effect.
2) Max requires a separate specular map defined in Materials when rendering. Since this is normally part of the data map, you end up having to pull the specular map out and saving it as a separate file to tell Max where it is unless the specular map is a 100% copy of your normal map.
3) Even if you manage to overcome 1 and 2, the render isn't going to look the same as it would ingame. In Sins, you can even use different color suns or skyboxes in order to achieve a reflective effect and show off your specular map. You can also see the effects of flair points or weapons fire, if you want to see these in the finished product.
So how do we get the model to appear like it would ingame? Simple: put it ingame! The developer executable allows you to replace the skybox with a greenscreen. That means you can put your ships ingame, take a screenshot without the HUD and icons, then import the screenshot, remove the greenscreen, and get a fully anti-aliased, anisotropically filtered, huge version of your ship on a transparent background, just as you would with a render, but with Sins effects applied.
Creating your unit picture
After starting the developer executable, hit Ctrl-Shift-Period ('.') to open the console. Hit the letter for Rendering (I think it's 'e') and enable greenscreen ('b'). Although this won't remove planets and asteroids, the former can be dealt with by moving your ships far away, and the latter can typically be dealt with through editing. You might also want to crank up your settings to maximum, enable 4x AA, and force anisotropic filtering at max using Catalyst Control Center (ATI) or nVidia Control Panel (nVidia). This can really enhance the look of ships facing away from you, whereas without AF, the textures would look blurry.
Position your ship or ships as you wish them to be shown in the icon. Zoom in as far as you can, turn on Cinematic Mode (F10 -> Effects -> Cinematic Mode), and then turn off the HUD (Ctrl- Shift-Z). I'm not sure whether zooming in here and reducing in Photoshop is a better idea than zooming out in Sins and then only slightly reducing in Photoshop since the game is essentially a vector reduction and Photoshop's is raster, but since your ships will be at a fraction of their original size, the quality difference shouldn't matter.
Take a screenshot. Make sure no planets are in the way, and try to reduce the asteroids in the shot. You can always edit these out later, but proactively minimizing the amount of work you need to do is good. Also, move your ships around if you want them to reflect the sun, planet light, etc, and customize the skybox through the console as needed. For example, I like to use a red skybox on my Soviet units, and a green skybox on my German units, this brings out their reflectivity and adds a nice shading effect.
Get rid of that greenscreen!
Okay, now tab into Photoshop and hit New. The dimensions of the new image in the dialog should be set exactly the same as the resolution of the Sins window or your monitor: Photoshop will usually detect your clipboard size and do this for you. After doing this, you can Ctrl-P to paste the clipboard image in. Right click the layer in the Layers dialog and hit Background to Layer so that you can get transparency when you delete the greenscreen.
There's a few ways to do this next part. I recommend using the Magic Wand - it's simple, usually effective enough, and using things like Color Range give you ugly and annoying green halos. Set Tolerance to 10 to 15, and click Anti-Alias - this will reduce the jagginess of the selection. Click the green background, and everything but your ship should be selected. Hit Delete to delete the green. You may be left with stuff that isn't green and isn't your ship, so you can delete those with the appropriate tools (Eraser, Lasso, Magic Wand, etc). You may also have green stuff between "cracks" in the ship - you can Magic Wand these and delete them as well.
At this point, you should have a pretty huge version of your ship with a transparent background. Awesome!
Editing in your picture
The next part is much easier. Copy a version of unit_picture.dds to your mod's Textures directory. We're going to assume here you want to replace a base Sins unit - you can also throw your picture into one of the empty spots, but this requires you change some things in the entity file and window file to make sure it's pointing to the right picture.
Open the new copy of the DDS. Select a empty box (one with just the background pattern on it with no ships) and copy and paste it - this should give you a new layer with the blank box. Keep this around - in fact, you might want to call it "blank" (any more blank boxes you do can be merged into the background layer as you only need one around for reference). You can go ahead and drag this new empty space on top of the picture you want to replace. Now you have a blank spot there in a separate layer (make sure it's on top of the existing background: you might also want to right click the background and hit Background to Layer to make deleting stuff easier on this layer).
Open the ship picture you just made and copy/paste it into your new unit_picture.dds. It will be absolutely massive. Don't worry! Make sure you have the new picture layer selected and hit Ctrl-T for free transform. Try to position the middle of the picture layer (the little grey circle) at the middle of the empty box you just created. You can then hold down Shift-Alt (to preserve aspect ratio and to make the layer scale toward the center instead of the upper left) until it fits in the box. If you have trouble doing this, try rearranging the ships by editing them before you transform (dragging fighters around, for example, so that they're closer to each other). Try to make the ships as large as possible while still fitting in the box: a little overlap on the grey stuff is OK, just don't overdo it. The exact size should be somewhere around 165x75. You can also drag the transformed layer around using your mouse. This comes in handy while the ship's picture is still huge and you can't find the center of the empty box: transform it down, don't complete the transformation, then drag the circle to the middle of the box and drag the transform back out. As long as you stay within one transform (don't hit Enter!!) and don't forget to hit Shift (to preserve aspect ratio) you can do as much resizing as you want without affecting image quality.
So now you should have your ship or ships in the box, transformed, and ready to go. Hit Enter to complete the transformation. Rename your layer to the name of your ship for future reference and then save it as a psd.
Making your alpha channel
One thing it would be easy to forget is your alpha channel. If you don't fix the alpha channel, you'll have a glow approximating the outline of the ship you replaced on top of your new ship. This isn't good. Hit the Channels button and toggle the visibility of the Alpha channel (hit the eye next to it) and you'll see the offending alpha channel blanked out as a silhouette of the old ship. We want to make sure we don't mess with any other alpha channels, so select a box around the old ship's silhouette, then use the Brush tool to fill this in until it's completely black (or red, if you have RGB still visible). Make sure you don't have the RGB channel selected while filling this in, just the Alpha one, or else you will be drawing on your actual ship rather than the alpha channel.
After this is finished, you can cut out your new ship. Toggle the visibility of the alpha channel again, go back to Layers, hit the Magic Wand, and click outside of the ship. If this is done correctly and the new picture layer is selected, it should select everything but your ship (all the transparent sections). You can make the ship layer the only thing visible to make sure, and make sure that "Sample All Layers" on the bar is unchecked. Then hit Select -> Inverse to select only your ship. Go ahead and make every layer visible again, then go to Channels and make sure every channel is visible. If you have just Alpha visible, this next part won't work, for some odd reason. Select the Alpha channel in Channels, then select the image window, and then hit Delete. Everything currently selected (read: your ship) will be deleted from the alpha channel, creating a silhouette just like the old one, where everything else is red but your ship. This means that the game will only do its light blur bloom thing around the outside of your unit. You can also blur this channel if you want to better approximate the original game's way of doing it, but I think the sharpness is fine.
Okay, if you figured that all out, you now have a blank layer, a layer with your ship sitting on top of that, and the shape of your ship is cut out in the alpha channel. You're pretty much done, but if you try to save to DDS, you'll get an error about "too many channels". This is kind of misleading.
Saving your new picture(s)
Save your project as a PSD one more time because the next thing you do is pretty much irreversible if you close Photoshop. Then right click any one of your layers and hit Flatten Layer. This will merge all your layers into one, assuming you have them in the right order (background -> background pattern blank box -> your ship on top) and the final product should look just like your ship was always there. Save this as a dds using the default settings as Unit_Picture.dds in the Textures folder, and you're done. Assuming your entity and window files point to the one you just replaced, your ship should now show up in the viewscreen. Every time you do this, it gets easier, and you can easily do three or four new ships in about a half hour or less once you've got it down. You can use the alpha channel trick for any other Sins dds that requires a alpha channel that's precise, too.
Hope this helped - I'll probably add pictures to make it easier to understand, but that's how to import a proper viewscreen picture into Sins for your mod.
Here's a few pictures I've made using the greenscreening process for Dawn of Victory in the process of making unit pictures.