Major Stress Major Stress

[MOD] The Sins Optimization Project version 1.0 (updated 1/25/15)

[MOD] The Sins Optimization Project version 1.0 (updated 1/25/15)

Because all good things must come to an end.

It is difficult to explain the issues of Sins have without going into too much "technical detail" which some may not understand. The main problem is that Sins is an older generation 32 bit game. With a two gigabyte memory limit. This is nothing new. All 32 bit games have a 2 gig memory limit. However Sins is NOT like other 32 bit games. Most games have an ability to load their assets "on the fly", or only as needed. Some games can also unload assets that are not needed. This is how they prevent themselves from reaching the 2 gig limit. Some games have been patched to where they can use up to 4 gigs of ram on 64 bit systems.

Sins also can load only what is needed at game start up, but Sins does NOT unload what it does not use. Any assets that are loaded remain loaded throughout the entire game! If you use all three factions (Six in the case of Rebellion), Then Sins will load ALL of its game assets at start up. Entrenchment, and Diplomacy in some games loaded between 1.7 to 1.9 gigs worth of assets at game start up! Graphic settings, The size of the Map, and the amount of Players, or AI opponents also determine how much ram Sins will use in any given session.

In 2006 when Sins was first conceived 32 bit Windows XP was the standard OS of the time. Single core CPU's were also still the standard. Though dual core CPU's were starting to become mainstream, and 2 gigs was considered to be "plenty" of memory. So most game developers at the time didn't have to worry about hitting that dreaded 2 gig brick wall. As time went on Computers, and Sins itself evolved into their present state which is now Rebellion, and soon to be Windows 10 running in multiple core 64 bit systems. With 16+ gigs of ram. Things have changed so much over the years. However Sins still remained the 32 bit game as it was designed in 2006. That uses only a single core of your CPU. With a two gigabyte hard coded memory limit. Sins WILL CRASH when the game reaches that 2 gig limit.

There is some misunderstanding about the 2 gig crash issue. The dump happens when Sins reaches 2 gigs of Ram Usage. That means the ram that Sins uses by itself. NOT what your entire operating system, and/or other programs you are running combined uses. Only the ram that Sins uses. No matter how powerful your system is. If it is 32, or 64 bit, or if it is Windows XP, Vista, 7, 8, etc. etc. Sins WILL CRASH when it hits its 2 gig ram limit. It is hard coded into the game.

Nobody was aware that this would be a major problem back when the game was first designed, because Sins was designed based on the hardware standards of 2006-2007. Before 64 bit OS's, Quad Core cpu's, and 4+ gig ram systems became the standard that they are now. There is nothing that can be done about it. Except to re-write Sins in 64 bit. Which is out of the question, because it is too cost prohibitive for the developers to do this (Ironclad said so themselves). Sins is a great game, and it is a shame that this is happening. With the Entrenchment, and Diplomacy expansions Ironclad had unintentionally pushed the game over its own hard coded limitations with all of the new added content.

Over the years as of the release of Diplomacy 1.3, and Rebellion. Stardock, and Ironclad did indeed fix all of the major game crashing issues of Sins. Using the fix's from This Project as a guide. However, NONE of these fix's ever made it into Entrenchment, or Original Sins! While the Stardock, and Ironclad developers have fixed most of these issues over years. They can not do anything to change the game itself short of reprogramming Sins over from scratch. The developers can do no more, but there is something that WE can do as modders to prevent Sins from hitting that dreaded 2 gig brick wall, and improve the performance of Sins "just a little bit more". I have taken the initiative, and applied to Sins what i have learned from my Homeworld Modding days. I present to you....

 

The goal of this project is to improve your Vanilla Sins gaming experience, and to eliminate the 2 gig crash issue. No changes will be made to Original Sins, or any of its Expansions in game play at all. No stat changes will be made. Nothing "new" will be added. There are other mods that can do that. Original Sins will remain Original Sins, and the Expansions will remain as they are. What we are doing is taking what game assets that Sins already has, and we are "optimizing" them to improve the performance, stability, and to moderate the ram usage in game. The side benefit will be that some lower end systems can enjoy almost the same gaming experience as the high end users do (Especially Laptop Users).

The project initially started out as an idea to reduce the texture sizes, and poly counts of some models. Not just arbitrary reductions. The goal was to maintain the quality of Sins in game. To make it look as if nothing has been changed at all.

We started with optimizations to the Planet's, and Skybox's, because they were the "worst offenders" as far as ram usage, and performance was concerned. Unfortunately my modeling skills left much to be desired in Softimage XSI at the time, and there were un-fixable mesh errors on the optimized planet meshes. Despite that the "proof of concept" was established. A lot of ram was saved, and there was a very noticeable increase in performance.

Then we studied the Particle files, and Textures. Many of the particle textures in our opinion were way too high in resolution for the purpose that they served. A reduction in texture resolution by HALF showed no visible difference in game at all! There were some exceptions where the particle resolution reduction difference was very noticeable in game, and those textures were reverted back to their original resolutions. Remember the goal was NOT to reduce in game quality. Also all of the particle textures were changed from TGA format to DDS format, because DDS uses a little less memory, and is much more efficient, because DDS format loads directly into video memory. The particles can also utilize the mip mapping features of DDS format. Where TGA format does not. Therefore enhancing in game video performance (increased FPS). As of version 0.07 no TGA format textures load in TSOP. With the exceptions of the Mouse Cursors, and Scenario Pictures. This is due to those textures being controlled by the hard coded exe. When you change the texture format those textures do not show up in game.

The developers mentioned something about a Memory Leak. I heard that a memory leak can be caused by a file searching for something over, and over again if it is not there. Example a mesh file calling for a texture that does not exist. Vanilla Sins had MANY particle, and mesh files that had that issue. We removed all of the entries that pointed to non existent textures in the mesh, and particle files. We also removed the entries that have lines like for example "C;\g\main\datasource\textures\effects\aura4.dds", and just used "aura4.dds". This will default the search to Sins, or the Mods "Textures" folder, and speed things up a bit. Also various typo's were corrected in some mesh, and particle files. Some ships had mesh nulls that were Miss-Labeled. For example the Akkan had nulls, and flairs that were miss-labeled. The engine trails, and flairs didnt show up in game. the Marza was missing its bomb nulls. etc. etc. All of this has been corrected as well. If there is a ships mesh file in the mods "Mesh" folder then it had something wrong with it that was fixed. The Soundata, and Galaxydef files had errors too. Regardless if any of these issues caused any memory leaks, or not these issues MUST be fixed! (and they were as of Diplomacy 1.3, but NOT for Original Sins, or Entrenchment).

After the Meshes, Particles, Textures, and the User Interface were optimized, and the very obvious issues were fixed there was a tremendous boost in game performance. Sins ram usage was greatly reduced. As of TSOP 0.07 I could run a game of Entrenchment, or Diplomacy ALL DAY, and I NEVER used more than 1.5 gigs of ram! This was using Maximum Settings. Including Bloom enabled. Tested on Huge Random Multi Star maps with 9 AI's. The 2 gig crash problem has been solved! Our primary goal with this project has been achieved! Without having to use any external memory increasing programs! Stardock, and Ironclad took notice, and they used TSOP as a guide in the Diplomacy 1.3 patch, and for developing Rebellion.

Now that the Stardock, and Ironclad have fixed the issues of the game as of Diplomacy 1.3, and Rebellion. TSOP has been focusing more on Original Sins, and Entrenchment. Since the Ironclad/Stardock fixes never made it into Original Sins, or Entrenchment. However, That is not stopping us from optimizing Diplomacy, or Rebellion "just a little bit more" ;)

You can also use Large Address Aware to give Sins an extra gig to play with, but in vanilla Original Sins, and Entrenchment the Endless Search Loop issues that were never fixed will cause the game to eventually go over that extra gig (Run vanilla Entrenchment, and watch your ram usage keep going up, and up, and up). TSOP is a solution that will stop that issue.

____________________________________________________________________________________________________________________

Downloads

The Sins Optimization Project v1.0

For Original Sins 1.195

Sins Optimization Project 1.0 for Original Sins


For Entrenchment 1.055

Sins Optimization Project 1.0 for Entrenchment


For Diplomacy 1.37

Stardock, and Ironclad fixed all of the issues that compelled us to make TSOP. Therefore there is no TSOP for Diplomacy


For Rebellion 1.82

As with Diplomacy, Stardock, and Ironclad fixed all of the issues that TSOP pointed out, and then some in the case of Rebellion. Sins: Rebellion is as optimized as any Sins game is going to get.



The Sins Optimization Project for Rebellion

The Sins Optimization Project for Rebellion

NOTE: TSOP 0.07 is still hosted on ModDB, and it still works in the current versions of Trinity (OS, E, and D), but you will be playing as if Sins was patched to an older version of the game. Plus some Icons and Buttons wont be visible.


Changes in TSOP 1.0

TSOP 1.0 updated to the latest versions of Original Sins, and Entrenchment.

The User Interface is completely overhauled to use pure DDS format.

Trade Ships,  Strike Craft, Mines, and Constructor Unit Meshes, and Textures are Optimized.

 Starbase Textures  are optimized mimicking the Stardock Rebellion Optimizations.

Sounds, and Music are optimized.

All TGA format Textures are replaced with DDS format, and TSOP "force" loads them. The only TGA textures that remain are the Mouse Cursers, and Scenario Pictures. This is due to hard code in the EXE.

 "Strategic Texture Reductions" for the Particles return using newer methods to create a better looking texture.

(Original Sins, and Entrenchment)  Many Mesh Null issues, and various File Errors were corrected, and Unnecessary Text Entry's were removed in both the Mesh, and Particle files. Entry's pointing to Non Existent Textures in both the mesh, and particle files removed. See the various "Fixed Files" texts in the mods Read Me for details.

(Original Sins, and Entrenchment)  The Diplomacy 1.3  graphic fix's have been incorporated into the Original Sins, and Entrenchment versions of TSOP (Pipeline Effects, "White Line" error fix, and Corrected Mesh files).

 TSOP 1.0 for Trinity will see the return of Optimized Planets using the planet meshes from Rebellion! but with our Optimized Original Sins Textures!

There will be no version of TSOP made for Diplomacy, or Rebellion, because there is no need for it. Only Original Sins, and Entrenchment need TSOP, because the 2 gig issue, and various other issues were never fixed in those two versions of the game.

This is the FINAL version of TSOP that i will make. If the community wants to take over the project they are more than welcome to.

 

Community input, and contributions are more than welcome, and they are encouraged! This is a Community Project! All can pitch in!

Note: TSOP does not fix the known issue of "Late Game Lag". It does help prolong the lag until further into the game in some cases, but sooner, or later the lag will happen. The late game lag issue is a CPU issue due to the fact that Sins (including Rebellion) is optimized to use only single core processors. Therefore Sins will use only one core in multi core CPU's. To help with the late game lag issue we recommend the following:

Playing on smaller maps.

Using fewer AI opponents.

Disabling the Trade Ship icons (this actually helps out a lot)

Stacking the icons in the empire tree (same as with disabling trade ship icons the less icons rendered the better).

Using a Strike Craft Reduction mod (The many strike craft in the game are a big CPU hog. There are some good strike craft mods that reduce the number, but maintain the balance. Search for them).

You do not need to download all versions of TSOP. You only need to download the version of TSOP for the specific version of Sins you wish to play. The Retail version of Trinity is just all three versions of Sins bundled into one package. Installation for Trinity versions of Sins are no different than if you purchased each version of Sins separately.

_____________________________________________________________________________________________________________________

Other mods are more than welcome to use TSOP. No questions asked! In fact we encourage it! Learn from it! Use it as guide, and an example! Merge it into your projects!

TSOP is intended to be a Stand Alone mod. Run with no other mods enabled. There are issues enabling TSOP with other mods. Enabling with other "Graphic Enhancement" mods is NOT recommended, because the graphic enhancement mod will defeat the entire purpose of TSOP. All of our optimized files will be overridden by that enhancement mods "enhanced" files. Increasing your ram usage instead of decreasing it. Some enhancement mods like Distant Stars are incorporating elements of TSOP into their future updates.

Read the included READ ME text's for installation instructions.

2,239,089 views 1,134 replies +10 Loading…
Reply #676 Top

Quoting myfist0, reply 675
I said I would import to XSI and remake the mesh file but I got no response. 
End of myfist0's quote

 

hmm sorry to hear that, who did you need a response from XD?

Reply #677 Top

Quoting Thahat, reply 674
huuuge radio silence here.

soo, did anyone have those landmines? are they easy to merge with the main mod?
End of Thahat's quote

That silence is called having real life jobs that take up most of our free time.

MyFisto it looks like it is on you to get the mines in game.. Time is a luxury i just do not have at the moment.

Reply #678 Top

Well, who is going to give them to me?

Reply #679 Top

Sorry, been offline for a little while. 

I uploaded the mine files HERE.  They are in .max format.  You should be able to import them.  If not, let me know.

 

SivCorp

Reply #680 Top

Thanks Siv, Im on it. XD

EDIT: I got them into XSI no problem but thats it and I have no clue whats messin it up. I can import any SoaSE model and export to mesh no problem. Heres the original TEC mine which I convert to mesh easily..

When I import Sivs and set it all up the converted mesh is well

TXT
MeshData
hasValidTangents TRUE
BoundingRadius -1.000000
MaxBoundingExtents [ -340282346638528860000000000000000000000.000000 -340282346638528860000000000000000000000.000000 -340282346638528860000000000000000000000.000000 ]
MinBoundingExtents [ 340282346638528860000000000000000000000.000000 340282346638528860000000000000000000000.000000 340282346638528860000000000000000000000.000000 ]
NumMaterials 1
Material
DiffuseTextureFileName "ModuleTechDeepSpaceMine-cl.dds"
SelfIlluminationTextureFileName "ModuleTechDeepSpaceMine-da.dds"
NormalTextureFileName "ModuleTechDeepSpaceMine-nm.dds"
DisplacementTextureFileName ""
TeamColorTextureFileName ""
Diffuse ffffffff
Ambient ffffffff
Specular ffffffff
Emissive ffffffff
Glossiness 50.000000
NumPoints 0
NumVertices 0
NumTriangles 0
NumCachedVertexIndicesInDirection:UP 0
NumCachedVertexIndicesInDirection:DOWN 0
NumCachedVertexIndicesInDirection:LEFT 0
NumCachedVertexIndicesInDirection:RIGHT 0
NumCachedVertexIndicesInDirection:FRONT 0
NumCachedVertexIndicesInDirection:BACK 0

 

Never seen nothing like it on all my conversions  X(  Heres a screen of Sivs setup.

 

Reply #681 Top

Quoting Major, reply 677

Quoting Thahat, reply 674huuuge radio silence here.

soo, did anyone have those landmines? are they easy to merge with the main mod?
That silence is called having real life jobs that take up most of our free time.

MyFisto it looks like it is on you to get the mines in game.. Time is a luxury i just do not have at the moment.

End of Major's quote

 

sorry if i offended, i did not mean to, i know it takes a lot of time and such, work/school etc. ( will start school myself on monday, figures. )

but it looked like it to me that that something was wrong or something, thats why i was wondering. did not mean it as a 'move on people /pompous voice'

so appollogies again!

 

@myfist0: is that a GOOD thing or a bad thing XD? i looked at the screenshots and, the spikes were gone, and the circles on the sides are diamond shaped now. didnt look half bad if you ask me XD, more tech-y, more squire XD

Reply #682 Top

The mine looks great but I cant get it to convert, need someone with more experience. The Vasari one has 3 sets of UVs, non of which are connected to the mesh. I gave up there.

@Thahat: no worries mate

Reply #683 Top

Dunno if it even has relevance(complete shot in the dark that will make me sound very windows for dummies =O) you left the optimized mine as Module-TechDeepSpaceMine instead of PlanetModule-TechDeepSpaceMine. Could it be something as tiny as a file name change messing up the conversion? =X

Reply #684 Top

I will double check the model in 3D Studio Max, and make sure that it isn't a problem within the model and get back to ya....

:waaaa:

 

Edit:

I checked the advent and Vas. Mines, and they all appear fine.

The Tec mine had some separation between a few mesh points, and has been fixed.  Here is the new download for you Myfist.

As to the nature of the problems your having...  I have no Idea.  If I had more time, I would have taken some of the load off of Major myself, but unfortunately I'm in the same boat, the boat with no time left. :annoyed:

+1 Loading…
Reply #685 Top

The problem is Mod Tool. Let me guess the error when you export is "cannot convert yada yada.dds in desired format, and resolution".. It is not exporting the textures. That is why i stopped using mod tool.

XSI 7.5 AND Mod Tool (built on the 7.5 foundation) will not save the tangents correctly. I get wierd lighting errors which others have had similar problems before. It has to do with the tangents. The model will seem fine in XSI, but when you convertXSI and test it in Sins it seems only a few polys that face the light source are glowing very bright (you can see the triangles on the mesh glow extremely bright, and it changes triangles as the ship rotates). While the rest of the mesh seems normal. I would post a screenshot if i was on my main puter right now to show you. Perhaps later. Setting has tangents to false corrects the issue, but that is something we really dont want to do. Someone mentioned how to fix this issue before, and the export problems from mod tool issue as well, but that was so long ago i forgot, and i cant find the posts here.

This is a very frustraiting issue. Perhaps convertXSI has issues with meshes made with XSI 7.5. I never had that kind of trouble with XSI 6.0, but 6.0 doesnt run on Vista, or Windows 7. I will not downgrade my systems, or get "ultimate" yet just to run one freaking modeling program.

If it wasnt for this the mines would already be in game.

If other XSI users see this perhaps you can shed some light on the problem?

Reply #686 Top

So is there a work around?  This really shouldn't be so complicated.... >:(

Reply #687 Top

Thats strange, I have never had a problem getting the textures to the mesh. If you look at what that mesh did export, the only thing it did correctly was THE TEXTURES.

Reply #688 Top

Unfortunately it IS that complicated for me. I am a 3dsmax user. Not an XSI user. Everything i know of XSI i learned from the Sins modding documentation. I cant build a mesh in XSI.

That is weird, because texture exportation is the only thing that doesnt work for me in Mod Tool, BUT the one mesh i did export had the very same tangent issue.

This is the SoA2 Klingon VoDleh, or "Emperor" class. It is the exact same model that is in SoA2 for Entrenchment. The Entrenchment model has no issues at all. It was exported with XSI 6.0. The only thing i changed for the Diplomacy version was the models scale, and moved the mesh nulls to match. This was all done in XSI 7.5

1st the tangent map. Done exactly how the modding documentation says. UV-ed from the top down gives me the best results (just like the advent ship used as an example). Everything looks just fine in xsi

However after i export to .xsi, and use convertXSI this is the result....

On top of that now it seems that ether XSI 7.5, or convertXSI is DELETING FACES ON ME!! As you can see on the torpedo tube.

 

XSI users can you throw us a freaking bone here? As you can see this is frustraiting me to no end. If i have to downgrade back to XP to use 6.0 again i am so done with sins modding.

 

Reply #689 Top

Can you send me the scene? 

BTW I found what was messing up the optimized planets.  :D  What was your goal there, smaller texture sheets? Like I said before though, I didn't really like the mesh decrease. Planets looked to blocky. Built from lego.

If the 3 texture sections have just "" you may have saved the model to dotXSI using version 3.5.

 

 

Using version 3.6 will give a few error pop-up messages but just click OK for each and the texture sections will be fill in.
 

 

 

Reply #690 Top

(facepalm) I clicked the "x" DOH!! OK i will give Mod Tool one last chance, but if it fails to export the tangents right then i am at a total loss.

I wonder if that was the issue with 7.5 as well asci 3.6 huh, ok let me test it. EDIT: Nope, failure. tangent issue still persists with 7.5.

Reply #691 Top

TXT
MeshData
hasValidTangents TRUE
BoundingRadius -1.000000
MaxBoundingExtents [ -340282346638528860000000000000000000000.000000 -340282346638528860000000000000000000000.000000 -340282346638528860000000000000000000000.000000 ]
MinBoundingExtents [ 340282346638528860000000000000000000000.000000 340282346638528860000000000000000000000.000000 340282346638528860000000000000000000000.000000 ]
End of quote

I have come across this error before when trying convert a file exported to .xsi from SI 2011. I think it is something to do with the fact that the .xsi file exported from the newer versions has some extra data and are arranged a bit differently, so the convertxsi doesn't recognize the file properly. A work around for this I have found is to export to .obj, then import into Modtool. You have to do the tangents and rendering again unfortunately.

Perhaps someone who is good at scripting can craft a new convertxsi application that will eliminate these problems. Though really my lack of knowledge on whats going on here leaves me guessing as to what is actually the problem.

Reply #692 Top

So export models into .OBJ first, and then import into mod tools?  I can save these .Max files out to .obj if that would help... Let me know folks!  :thumbsup:

Reply #693 Top

 

 

Reply #694 Top

.OBJ saves the UVs and .FBX saves the mesh points.

Converted mesh to BIN imported to 3DS. Exported OBJ to XSI 7.5 and did the UVs and tangents as per the documentation.

EDIT: Fixed thanks to Zs post

Reply #695 Top

Hey Major Stress, when your model is in modtool, in the explorer how does your set up look alongside the blue box under clusters? does it say "Sample" or "Texture_Coordinates_AUTO". If it is "Sample" change it to "Texture_Coordinates_AUTO". You should then be able to do your tangents as described in the forge tools documentation & they should export correctly.

Another thing that might be happening IDK for sure, is that convertxsi sometimes bends edges the opposite way, especially noticeable on models that are symmetrical. Putting in a few extra edges around where this is happening can fix this.

Reply #696 Top

.OBJ saves the UVs and .FBX saves the mesh points.

Converted mesh to BIN imported to 3DS. Exported OBJ to XSI 7.5 and did the UVs and tangents as per the documentation.

1 issue is it is saving 2 sets of materials even after I clean the scene and freeze. If I delete the empty set from the mesh its a no-go.

NumMaterials 2
Material
End of quote

When you import and obj. into a scene you get the material(s) that come with it. This is on top of the material that already exist within the scene. Go to the materials manager, its @ > Render(3 on key board) > Modify > Material. the material manager should pop up. Now you need to right click on the SceneMaterial and in the box that pops up click "assign to selection", then pick your imported mesh in the explorer. The wire115115 or what ever should then be redundant and you can delete it.

You should be able to just delete the unused material in the explorer though, unless the obj. you import has more than 1 material.

[quote]If I delete the empty set from the mesh its a no-go.[quote]

I would say that all your triangles are assigned to material0 (the top 1 without the texture files) you could try shifting the texture files from the second material and delete the second material(material1)

+1 Loading…
Reply #697 Top

Klingon_VoDleh.mesh Test her out boys

Dropbox is temp share

That worked great Z.  k1  for teaching but the "Sample" or "Texture_Coordinates_AUTO" makes no difference as I have done without changing and it works fine.

This is from Creating the Tangent Map

Note: If your UVs were not created in XSI, the directory structure might be named differently.

Nothing about changing.

Here is the scene. It is the FBX import giving me issues, I needed to look at the FBX export settings again.

http://dl.dropbox.com/u/1514086/Klingon_VoDleh.exp updated with points

Reply #698 Top

Oh! maybe that was needed for something else, perhaps it was assigning extra material I can't remember. Anyway I am glad the problem is fixed.

In previous attempts at converting the file to mesh, did it succeed at writing in the point data? If so you could just copy them.

Reply #699 Top

Ya I copied all the mesh points. I did get the FBX to work, went back to default export settings and I didn't notice.

Importing the mesh points does mess up the names though so it's still easier to copy them.

I will revisit the mines this weekend.

Reply #700 Top

It seems it was as i feared xsi 7.5, and above, and convertXSI dont get along that well. Thanks for getting on this guys. I think i have the problem almost solved except the deleted faces.

I knew about the texture coordinates thing. Found out about it the hard way in the early days.

I managed to stop the tangent artifacts thanks to you guys help, but i still have the problem of the deleted faces in the torpedo tube. If not for that the mesh would be perfect. There is no way that i can see in XSI to check if a face is flipped. I tried re-importing the mesh from the master 3ds file. same result. The mesh looks fine in XSI, but you dont see the missing faces until after you import to sins. I have this same problem on the SoA2 Kvort, and the Excelsior, but on the excelsior the face that is deleted is so small you dont notice it. So i left that model alone.

Dont bother with the mesh nulls. i created a new set for the VoDleh. With nav lights ;)

EDIT: I give up on the deleted faces issue. after repeated attempts of re-importing the model i get the same results be it from .obj or directx format. Mesh is fine in 3dsmax, lightwave, and MS3d. Deep Exploration shows no errors ether. So this must be happening on the conversion to mesh.