BingXueZhiYa BingXueZhiYa

[Discussion]Some question about the "4GB patch" using for SSE

[Discussion]Some question about the "4GB patch" using for SSE

I have got a 4GB patch from the maelstrom mod(really a brilliant work) these days. The author of M said it would allowed the SSE to run highest opinion with his mod without minidump. 

I searched the developer web of this patch and found it will increase the virtual memory for a 32 program, I test it on my 64 OS Dell and it indeed work:

this is the memory situation before it crashed. The memory_doge is a easy program to apply memory unstoppable.

 

the bellow is the same program with the patch.

 

So, until now the patch works well, but when it comes to SSE, some things interesting happens:

I didn't took a picture on the game without this patch, I think it is no need as the memory use doesn't even exceed 2GB.

I took the picture when I started to feel the speed of the game is slowing down.

The normal game:

the number of entities, I didn't show the mesh situation as they will be released once you move your camera.

The game with the mod maelstrom:

If you find hard to understand Chinese, well, the last number is virtual memory and the number next to it is the memory that really loaded in RAM.

CPU : Core(TM) i7-4500U 1.80 GHz 2.40 GHz X 4

RAM : 8.00 GB

I discussed this problem with a software engineer and he told me that this program may hold its memory usage to not let it exceed 2GB by save some information on ROM and load it when needed( you did't mishear, this program are using the same method used by the CPU to arrange RAM space) 

What more, he tested and told me that the usage of GPU is still not optimistic. . . 


So, could there anyone tell me that how and why this patch is useful for SSE as someone including me found the times of minidump in maelstrom is REALLY decreased.

 

51,523 views 48 replies
Reply #26 Top

That only concludes that sins still will not go over 2 gigs. These "patches" may give sins the 2 gigs of "room" it needs in your system, but they will not force sins to use more than what is hardcoded into it. Kind of like trying to force a square peg into a round hole.

Other games like the X3 series were patched by the developers over time to use more than 2 gigs of ram. I believe like how it was explained before by making it use more than one process. TBH I am not exactly sure how they did it. However Sins is optimized to use only a single core 32 bit CPU. We have tried in the past to ask the dev's if what was done to games like X3 can be done with Sins. The answer was a loud resounding NO! I believe it was not that it couldn't be done. Just that IC/SD at the time didn't have the money, manpower, or resources to do it.

Reply #27 Top

Quoting Major, reply 26

That only concludes that sins still will not go over 2 gigs. These "patches" may give sins the 2 gigs of "room" it needs in your system, but they will not force sins to use more than what is hardcoded into it. Kind of like trying to force a square peg into a round hole.

Other games like the X3 series were patched by the developers over time to use more than 2 gigs of ram. I believe like how it was explained before by making it use more than one process. TBH I am not exactly sure how they did it. However Sins is optimized to use only a single core 32 bit CPU. We have tried in the past to ask the dev's if what was done to games like X3 can be done with Sins. The answer was a loud resounding NO! I believe it was not that it couldn't be done. Just that IC/SD at the time didn't have the money, manpower, or resources to do it.
End of Major's quote

After all, StarDock is just a small company working on office software.....And they are spending time on GC III these days I guess.

Reply #28 Top

Quoting soase-maelstrom, reply 25

Have been testing Rebellion (Malestrom) 1.82 with the "4GB patch".

Random Huge and Vast maps, 10 players, highest settings for all graphics effects.

Whereas normally Rebellion (without the patch) would crash around 1.7GB RAM usage with the '4GB patch' I can get stable operation to about 1.95 GB. I have had RAM usage peak just above 2GB before a crash.

PS: My System is as follows: OS is Win 7 64-bit pro, 8GB RAM, Intel i7-2600 @3.4GHz. Graphics is AMD Radeon HD 7900 with 3 monitors.
End of soase-maelstrom's quote

Just as you said, it didn't pass the 2GB peak......But just like Major Stress said, it may give the room if needed, probably this is the reason why it not crash like before. Loading and saving data on disk still need some RAM , right?

Reply #29 Top

Quoting WOEaintME, reply 17

From a mathematical point of view 2^32 = 4 GB  but generally the first bit must be zero resulting in 2^(32-1) = 2 GB.

The jump to 4 GB can be easily shown but the jump beyond 4 GB, while doable, is not something that can be easily done. This is obviously for a 32 bit program, 64 bit programs are essentially limitless (2^(64-1) = 2 billion GB).
End of WOEaintME's quote

The Windows kernel (or 32-bit subsystem imitating that [wow64]) is also mapped to those 4 GB for 32-bit apps. By default the split is 2G/2G user/kernel. With /LARGEADDRESSAWARE (/LAA for short) it's 3G user/1G kernel. Look in Windows Internals for example.

You can't just mark the exe as /LAA, there are some technical gotchas to test/prepare your 32-bit app for 3GB, which may need source code changes. Which is why devs don't spam this LAA bit willy-nilly. See http://books.google.com/books?id=wYrCitbs5PQC&pg=PT451 for instance.

 

 

Reply #30 Top

Quoting RespawnedTitanL10, reply 29


You can't just mark the exe as /LAA, there are some technical gotchas to test/prepare your 32-bit app for 3GB, which may need source code changes. Which is why devs don't spam this LAA bit willy-nilly. See http://books.google.com/books?id=wYrCitbs5PQC&pg=PT451 for instance.
 
 
End of RespawnedTitanL10's quote

 

Yes, either 4GB patch or LAA could work on the "memory doge" as this program doesn't have RAM control at all to avoid crash. But I guess SSE has a very 'strict' RAM control to hold the RAM below 2GB. Without source code change, all kinds of those patch will not work.

Reply #31 Top

Wouldn't compiling the game's source for an x86_64 target allow the game use more than 2GB? Even if Ironclad are busy with other projects, having someone go back and do a recompile would take a few hours at max (and it would be a recompile, since all x86 instructions are fully supported in x86_64), though I understand you'd still need to do QA and Steam certification. Maybe release it as a beta the way the 1.82 patch was for the longest time?

 
Reply #32 Top

Quoting Delnar_Ersike, reply 31

Wouldn't compiling the game's source for an x86_64 target allow the game use more than 2GB? Even if Ironclad are busy with other projects, having someone go back and do a recompile would take a few hours at max (and it would be a recompile, since all x86 instructions are fully supported in x86_64), though I understand you'd still need to do QA and Steam certification. Maybe release it as a beta the way the 1.82 patch was for the longest time?

 
End of Delnar_Ersike's quote

If we recompile it, most of problem about RAM will be end. But just like Reply 26 said, the answer was a loud resounding NO. Another question is if don't make the game two version (64 bit and 32 bit) the 32 bit will lose this game as the a 32 bit OS can not support 64 bit program.

Reply #33 Top

Quoting Delnar_Ersike, reply 31

Wouldn't compiling the game's source for an x86_64 target allow the game use more than 2GB? Even if Ironclad are busy with other projects, having someone go back and do a recompile would take a few hours at max (and it would be a recompile, since all x86 instructions are fully supported in x86_64), though I understand you'd still need to do QA and Steam certification. Maybe release it as a beta the way the 1.82 patch was for the longest time?

 
End of Delnar_Ersike's quote

Can you even make 64-bit DX9 games? All 64-bit games I know of use DX10 or later. That is proabably the biggest show stopper. The programming models of DX9 and DX10 are substantially different. Never mind all the other binary dependencies, bink etc.

Hmm, apparently Crysis (I) although always using a 64-bit core could switch between DX9 and DX10 at the video layer. So perhaps not impossible.

 

Reply #34 Top

Quoting BingXueZhiYa, reply 32
If we recompile it, most of problem about RAM will be end. But just like Reply 26 said, the answer was a loud resounding NO. Another question is if don't make the game two version (64 bit and 32 bit) the 32 bit will lose this game as the a 32 bit OS can not support 64 bit program.
End of BingXueZhiYa's quote

We wouldn't be able to recompile it, but Ironclad could. After compiling for an x86_64 target, you wouldn't need to delete the old, 32-bit .exe, just make both available and let uses with 64-bit OS's choose which .exe to launch (quite a lot of games do this, actually).

Quoting RespawnedTitanL10, reply 33
Can you even make 64-bit DX9 games?
End of RespawnedTitanL10's quote

You can. The Microsoft article does point out some possible problems though, so porting to x86_64 isn't as easy as I originally thought; it really depends on how Sins was programmed, but it definitely would take more than an hour.

 
Reply #35 Top

I'm going to revive the argument that LAA can let Sins use more than 2 GB of RAM, as it can in fact do so; I've managed to push Sins past 2 GB with mods and here's a picture from a few minutes ago. Usage is just over 2.1 GB.

Reply #36 Top

Apps do indeed benefit automatically from having /LAA added (as long as they rely on fairly standard memory allocation calls)... assuming they don't start crashing in new ways, like I said. The don't need changes to make use of /LAA's main benefit. "Porting" to /LAA is basically a series of manual (or at least some semi-automatic) code checks to be somewhat sure the app can handle the 3GB address space without running into certain kinds of pointer bugs stemming from 2GB addresses assumptions, e.g. assuming that all pointers are 31-bit and using the top bit for some special purpose, ensuring that pointer subtractions don't overflow when moving to 3GB because signed 32-bit integers don't fit the difference between extreme addresses in the 3GB space, etc. But I do suspect that most programs don't have the kind of weird pointer manipulation that would cause problems when using 3GB, so the MS guys may be too conservative with their (code auditing) recommendations...

Sins already crashes a lot and I think it's occasionally from its memory management, although not necessarily because it runs out memory; other kinds of mm bugs are possible. From what I see, I think only some part of Sins can really make use of the extra RAM granted by /LAA; the 10% gains (to 2.2GB) don't seem worth the hassle of risking new bugs, but YMMV as they say.

Perhaps some mods require the extra ram, but the stock game, even with 100 bomber squads on the screen, stays under 1GB for me, although it does start to lose fluidity when zoomed in (to see 500 individual bombers) but it's the CPU, not the GPU [or RAM] that's the limiting factor there.

Sins does benefit from a SSD more than most games; its loading times are pretty slow from a HDD.

 

Reply #37 Top

Quoting RespawnedTitanL10, reply 36


Sins does benefit from a SSD more than most games; its loading times are pretty slow from a HDD.

 
End of RespawnedTitanL10's quote

 

Can' t agree more, due to  the window show when SSE crash is "minidump" by the program instead of the "runtime error, oxXXXXXXXX can not be read only" or so on by OS. We could see it is not a RAM leak or limit error (just I guess).

BTW, some of my friends have tried to SSD for SSE, but the improvement is not obviously. Probably the count of calculation is the point as the algorithm in the game may be out-of-date or the problem itself doesn't have a better solution (Hamilton circuit for example)

Reply #38 Top

I would think the benefit of the SSD would be load times only, yes?  Unless you have a lot of the memory usage going to a pagefile?

Reply #39 Top

Quoting furyofthestars, reply 38

I would think the benefit of the SSD would be load times only, yes?  Unless you have a lot of the memory usage going to a pagefile?
End of furyofthestars's quote

Yes, Sins doesn't seem to cache much itself, probably because of address space limitation. I don't have exact numbers, but subjectively load times went from 10-15 secs with a HDD (startup and loading the 1st game) to 3-4 seconds with SSD. Most games don't see 3x-5x speedup with SSD.

 

Reply #40 Top

I think Sins does use the paging file because if it is too small, you will find textures won't load in the game, you just get the white squares....

Reply #41 Top

Quoting RespawnedTitanL10, reply 36
From what I see, I think only some part of Sins can really make use of the extra RAM granted by /LAA; the 10% gains (to 2.2GB) don't seem worth the hassle of risking new bugs, but YMMV as they say.
End of RespawnedTitanL10's quote

Do you have any evidence to show that Sins will only get 10% gains? My picture there is not a representation of the maximum Sins can go to with LAA; it is purely what that specific mod could do at the beginning of a match.

Quoting Seleuceia, reply 40
I think Sins does use the paging file because if it is too small, you will find textures won't load in the game, you just get the white squares....
End of Seleuceia's quote

I can confirm this. Several of my testers had issues due to lacking a paging file in the past, and Sins started working without issue for them once it was up and running.

Reply #42 Top

Ahh, ok.  For the longest time I thought the PF was simply an "extension" of sorts of the normal RAM and that it was only really used once the physical RAM was starting to run out (within the limits of what the OS could support, of course).  I didn't realize individual applications could not only uniquely identify the PF space, but could explicitly cache things into it.

Reply #43 Top

No, you can't explicitly cache anything into the PF (from an application). It's up to the OS to page stuff out from RAM, in which you can indeed cache stuff. People who see PF usage increase/pressure with Sins probably have too little RAM (for OS + game + video if not using dedicated VRAM). My gaming box has 12GB and the PF is set to 4GB (more would be a waste of SSD). It hardly ever gets touched even then. About 160KB of Sins ever gets paged out, after a couple of hours play. Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx) is good for finding out stuff like this.

 

Reply #44 Top

Quoting RespawnedTitanL10, reply 43
People who see PF usage increase/pressure with Sins probably have too little RAM (for OS + game + video if not using dedicated VRAM).
End of RespawnedTitanL10's quote

If your paging file is ~2GB or less, overtime your Sins game will be unable to load textures...you could have all the RAM in the world, it wouldn't matter....large mods run into this issue faster, but it can happen with vanilla, and even for users who have large quantities of RAM (basically anything 6GBs or bigger)...

Reply #45 Top

Quoting Seleuceia, reply 44


Quoting RespawnedTitanL10, reply 43People who see PF usage increase/pressure with Sins probably have too little RAM (for OS + game + video if not using dedicated VRAM).

If your paging file is ~2GB or less, overtime your Sins game will be unable to load textures...you could have all the RAM in the world, it wouldn't matter....large mods run into this issue faster, but it can happen with vanilla, and even for users who have large quantities of RAM (basically anything 6GBs or bigger)...
End of Seleuceia's quote

Sounds more like a bug in your graphics driver [version]. Nvidia has been very problematic for me in past couple of years. I'm basically stuck at version 314.22 of their drivers for my GTX 560. Other versions (older or newer) have had one problem or another, from bsods to inappropriate power management, to the driver not releasing GPU memory properly when OpenCL programs finished (with flaccl in particular) and the list goes on... I do remember that another DX 9 game I enjoy, Supcom FA, has had problems with textures starting to corrupt after playing for a while, and more likely to happen when I alt-tabbed out of the game and back. But that was with an older driver IIRC, although I haven't played supcom FA in a while. (The MP matches there are also role-based [like the 5's] but more fun that Sins' IMO.)

Also, I suppose you know you can configure the paging file with both a min and max. Did Windows really fail to grow your pagefile when it actually needed to do so or did you set the max to 2GB as well? (On my box, the max is set to 16GB, although I've never seen the pagefile grow beyond the min.)

And since you mentioned this was with a mod: the GPU can also use your [main] system RAM for textures [and, I suspect, eventually/indirectly the page file] if it runs out of on-board video memory for that. Checking out GPU [on-board] video memory usage is possible with Process Explorer as well. Sins uses about 512MB video RAM with its standard textures. But a mod can probably well exceed that if it wasn't profiled to fit in the [typical consumer] video card. The graphics drivers in a 64-bit OS (since you mentioned 6GB of RAM, I assume that was the case) can use an obscene amount of virtual memory for textures. I'm not sure what memory usage limits DX9 places on textures loaded from a 32-bit app in a 64-bit OS. The MS docs are a bit hazy on this (and I'm no gfx programmer), but it sounds like once you push the texture to DX9's texture manager (http://msdn.microsoft.com/en-us/library/windows/desktop/bb172341(v=vs.85).aspx), you can free the in-app memory for it, so in theory a 32-bit app can push lotsa textures in a 64-bit OS and it's the texture manager's problem to juggle them in and out of video RAM.

Update: Apparently it's not possible to exceed the process virtual address space (2GB for Sins) when pushing textures to D3D9... unless one bypasses D3D9 texture management layer: http://gamedev.stackexchange.com/questions/68270/is-there-a-way-to-limit-the-size-of-the-d3dpool-managed-pool. I'm having doubts Sins does that, but who knows... And to venture one final guess here: the D3D9 texture manager most like handles /LAA (3GB addresses) correctly, so I think Sins would get more texture memory (up to one more GB) when ran with /LAA, assuming Sins uses D3DPOOL_MANAGED textures.

 

 

Reply #46 Top

Quoting RespawnedTitanL10, reply 45
Sounds more like a bug in your graphics driver [version].
End of RespawnedTitanL10's quote

I've played Sins since its release and have played large mods since around entrenchment...over that time, I would have used several ATI CCC versions for an AMD card on a Dell with Vista and eventually W7 as well as several NVIDIA driver versions for a NVIDIA card on a Lenovo with W7, W8, and W8.1...2 totally different graphics cards, computer manufacturers, dozens of driver versions, and 3 operating systems...I would be very surprised if the issue is a driver, because that is a lot of drivers to be at fault...

Quoting RespawnedTitanL10, reply 45
Also, I suppose you know you can configure the paging file with both a min and max.
End of RespawnedTitanL10's quote

Quoting RespawnedTitanL10, reply 45
Did Windows really fail to grow your pagefile when it actually needed to do so
End of RespawnedTitanL10's quote

Well the default setting lets windows manage it and set whatever size it wants, which apparently isn't good enough for Sins...I've always had to go in after each reinstall and manually set it (I usually pick 4-8GB as my range) and that seems to fix the problem...

Before Rebellion, I only had paging file issues when playing really big mods, though for a while I just thought it was an issue with the mods...eventually I saw in a mod ReadMe to up the size of the paging file, and realized what had been happening...with Rebellion, I get it with the base game if I play several games without restarting the computer, so even without mods I'd still need the larger paging file size....of course with mods like SOGE, the larger paging file size is a must...

As a little aside, on my current computer if I manually set it, it seems to already be set with a max of 2GBs even though it recommends a min of 4GBs...go figure...

Way back in the day though, before Entrenchment, I played with a friend who always would have textures not load late in the game...he had lots of RAM, and remember this is original Sins so he should never have run out of RAM...I don't know if it was a paging file thing or not, but he experienced the same symptoms...

I don't know how Sins is programmed, but it definitely has memory issues....lots of skilled players restart their game after finishing a match on ICO to avoid crashing while in the lobby, and that's assuming they don't crash while viewing the end-game stats...a dev actually confirmed these MDs were simply related to memory...with all that in mind, the fact that manually increasing the paging file size fixes the issues I've experienced (as well as many others who have used mods), I'm inclined to think the issue is the game...I'm sitting here with 8GB of RAM and a video card with 2GB of its own memory, there's no reason Sins would make use of the paging file unless it was programmed to go that route...

Reply #47 Top

There were apparently lotsa gotchas with DX9 on Vista [related to memory usage/allocation], but I guess people don't run into those anymore. I found some interesting resources at http://blogs.msdn.com/b/chuckw/archive/2010/12/16/windows-x64.aspx but I don't' have time to read them tonight. Note that his presentation "Why Your Windows Game Won't Run In 2,147,352,576 Bytes" is linked on the bottom of that page (as zipped PPT "attachment").

 

Reply #48 Top

Two quick notes based on that presentation:

  • On 64-bit Windows /LAA actually gives the app the full 4 GB entirely for user space (the 1GB/3GB split is for 32-bit kernels) and this requires no special boot flags (unlike the 32-bit kernel case). As for testing your machine's LAA config: http://technet.microsoft.com/en-us/sysinternals/bb963901 has the "testlimit" utility. The 32-bit version of that utility when ran (with the -r flag) on 64-bit Windows 7 can indeed allocate 4032MB of user/app space; and if you have enough physical RAM, like I do, you can pass testlimit -d, which will also touch every page allocated, forcing the OS to actually provide the app with that amount of RAM (instead of just address space.) [I suspect "-d" would work just with enough pagefile, but it will thrash your disk/SSD like crazy.] Given that nobody posted Sins getting anywhere close to 4GB (when /LAA patched), there are probably quite a few limitations in the Sins code.
  • There are some 31-bit gremlins even in the Microsoft DX9 code, for example "D3DX9’s Effects system uses the 32nd bit. High-bit indicates a handle vs. a string. There’s a mitigation [in] DirectX SDK (June 2007) or later. Eliminate string usage for D3DXHANDLE. Create with D3DXFX_LARGEADDRESSAWARE." So for /LAA the effects code needs to be recompiled with that in mind; more at http://msdn.microsoft.com/en-us/library/windows/desktop/bb172855(v=vs.85).aspx. I've looked at the calls Sins makes to D3DXCreateEffectFromFileW, which it calls a lot on start-up. Alas, the flags Sins passes to that function are always set to 0x20=32 so never include D3DXFX_LARGEADDRESSAWARE (1<<17=131072), which is needed for proper operation under /LAA. And you cannot simply put a shim DLL in between to fix that; see the previous msdn link why not.

It appears Sins devs have investigated using LAA, but decided against it: https://forums.sinsofasolarempire.com/427540/page/1/