[Crash Fix] Pelaa-Sins

Another one huh?  You're the 4th or 5th person I know of that's 'helped' in this manner.  I applaud your efforts, but this doesn't fix anything, not how you're representing it anyway.  It also appears that you've been seriously misinformed or totally misunderstanding what's going on, not only with LAA but the issue with Sins crashing and virtual memory in the Windows environment...  Basically, everything you're talking about.  I'll lay down some education for you and those that read this so hopefully no one does anything foolish.

First about the Virtual Memory.

Sins doesn't crash because you run out of virtual memory.  Virtual memory size in Windows can be controlled and is usually controlled by the OS adding and reducing as needed, but can also be set by the individual using the computer. For example, I just turned on my laptop and brought up the screen on virtual memory usage.  Look at this screenshot:

http://sites.google.com/site/steelers086/gamefiles/VirtRAM.jpg?attredirects=0

The key things to notice are that it is vista service pack 2, it's a 32 bit OS, the virtual ram is system controlled, it recommends 4.6Gigs and allocated 3.3Gigs, right at start up.  Literally, I turned it on, opened up those windows and snapped the screenshot.  My 64bit system, the one I'm typing on right now allocates over 6Gigs to virtual RAM.  Therefor it's impossible to run out of virtual memory by running Sins since windows by default allocates more then what Sins can use right at startup.

Sins crashes because it runs out of memory period, virtual or physical or both.  All programs use physical memory until there is no more space, then the OS starts using virtual memory in the form of swap files to allow more programs to run by moving inactive 'running' programs to the hard drive and then pulling it back to RAM when it's accessed again or more space is freed up by you closing out other programs.  If at any point you get close to running out, as long as you haven't screwed with the setting of your OS, it will automatically assign more hard drive space to be used up in the form of swap files, thus increasing your virtual RAM size, and then reducing it back down when it's no longer needed.

By default a 32bit program can only address up to 2Gigs of memory.  Sins crosses this threshold quite easily and goes boom.  Virtual memory is not an issue to even be discussed anymore because...  Just because...

Now on to LAA (Large Address Aware).

This is a program flag that allows the program to recognize memory addresses larger then the assignable addresses that come standard to a 32bit system.  Do keep in mind that even changing this flag, you still have a 32bit program.  That means it can still only use up to 2Gigs of address space, period.  The only way to change that is by rewriting it as a 64bit executable.  That flag enabled allows it to use larger addresses, and effectively move itself out of the range of addresses that other 32bit programs use, allowing it to rely LESS on swap files and virtual memory, thus improving it's performance, which in turn gives off the false impression of fixing the game.

To read other discussions on LAA visit these links:

https://forums.sinsofasolarempire.com/335474/page/140/#2696581

Specifically points #'s 4 and 10.

There's also this one that references other times the issue has been brought up and my responses.

https://forums.sinsofasolarempire.com/335474/page/137/#2670665

If you don't want to read my responses, then feel free to read one from the game's developer that basically says the same thing as me, and the more detailed response just above his.

https://forums.sinsofasolarempire.com/375480/page/1/#2533254

Then I refer you to my first link and point #4.

Now onto Sins.

Sins with the help of mods, runs face first into the hard coded limit of 2Gigs.  To get to that point slower, the best advice and only solution we have available is to optimize the textures and/or run them at lower settings, or to get a 64bit executable made for it, or to run it on a Linux OS where it won't be restricted by the limits of the Windows environment (so I'm told).

LAA get's brought up all of the time in many different forms and fashions, explained in several different ways, and always incompletely or incorrectly.  The only constant claim about LAA is that it's not what people think it is and doesn't do what they're claiming.  To any reasonable person, if you had to pick between an argument that is constant and unwavering, or one that changes from person to person, which do you think is the more accurate argument?

My point is, thank you for your time and efforts to help out the community, however, your claim is inaccurate and misguided, and hopefully with a better understanding that I've tried to put out here, you, as well as others will turn your attentions to proper solutions rather then placebos.

This is bogus. Programs do not directly use physical memory. Ever. Period. That's the point of virtual memory! Virtual memory is an abstraction around the various discontiguous memory backends that the system has available; for example, the swap file, and the physical RAM. The only software on your system that is allowed to directly access physical RAM is the Windows kernel.

There are two senses in which the term "virtual memory" can be used: the process address space, and the system address space.

In the process address space of a 32-bit process, the virtual memory is an abstract linear array of memory locations. The size of this array is 2GB by default (without LAA); 3GB with the Large Address Aware flag on 32-bit OSes; and 4GB with the Large Address Aware flag on 64-bit OSes. If you're running a native 64-bit program, the size of the process's address space is 2 TiB.

Note that every single process on your system gets its own virtual memory, regardless of how much physical RAM your system has. So if I create 200,000 processes on a 64-bit computer with 4GB of RAM, each one of those processes gets its own, clean slate, 2 TiB of virtual address space. As you can easily see, 200000 * 2 TiB is much, much larger than 4GB. How is this possible? That's the very purpose of virtual memory: every memory allocation goes through the system allocator, which maps virtual memory against backends, such as RAM or the swap file.

This is also bogus. In reality, the operating system starts to swap out data (into your swap file) long before you actually use up all your physical RAM. The term is called "Swappiness". Low swappiness means that swap file is used more sparingly; high swappiness means the swap file is used more aggressively (i.e. even when there's physical RAM to spare). The swappiness varies, depending on the size of your RAM, the size of your swap file, and the version of Windows you're running. I find that at about 90% physical RAM usage, inactive processes start to get swapped out to disk, even though 10% of my physical RAM still remains. Of course, your mileage may vary.

Your account of how processes are moved back and forth from RAM to the swap file are correct, but you use the term "Virtual RAM", which is technically incorrect. At the userspace level, when a process requests RAM via HeapAlloc() or malloc(), it has no idea whether it's going to be allocated against physical RAM, a swap file, an SSD, a magnetic tape, an abacus, ... you get the point. By "Virtual RAM" I think you are referring to the total amount of available system virtual memory (not to be confused with the amount of virtual address space in a process).

Actually, virtual memory is at the very heart of the issue. The topic under discussion is the amount of virtual address space granted to user-space processes by the virtual memory subsystem.

This is also incorrect. First of all, what's 2 to the 32nd power? Type into google, 2^32. The answer:
4 294 967 296

Now, this number is the theoretical max of how much address space is available to a 32-bit program, because "32 bit" means "32 bits of address space". So you get 4294967296 bytes (or 4 Gigabytes) of address space.

Why, then, does Windows limit this to only 2GB?

Because the virtual address space has to be shared with the kernel. The kernel has to be able to access data in the process's address space in response to system calls. To obey its own rules of process address space separation, it therefore needs to reserve part of the address space for itself. Source: http://www.microsoft.com/whdc/system/platform/server/pae/paemem.mspx (the first and second paragraphs; substitute "Windows executive" with "kernel", since that's what it is).

This limitation doesn't occur on 64-bit Windows, because 64-bit Windows runs 32-bit processes in, basically, a thin virtual machine environment. The areas used by the Windows kernel reside in a 64-bit address space completely outside the 32-bit address space given to the process. Since the kernel doesn't need any space within the VAS of the userspace process, it just gives the full 4GB (2^32) to the userspace process.

To give Sins more than 4GB of virtual address space (or 3GB on 32-bit), the process would have to be recompiled for native 64-bit; yes. But you can fit considerably more content into 3GB or 4GB than just 2GB. If your mod is hitting out of memory errors with Large Address Aware on, your mod is either super mega insanely big, or you're fighting a ridiculously large battle, or your content is larger than it should be (wasteful). But we can all agree that with just 2GB of VAS, battles don't have to be *that* large, and mods don't have to be *that* complex; it will crash with out of memory fairly often, leaving much to be desired. I have no way to give you the insane amount of VAS that a native 64-bit executable would enable, but I can slip you a few more gigs.

1. Optimizing stuff / running at lower settings: This IS helpful. I fully agree that the Sins Optimization Project is HELPING. Maybe for some content, just integrating TSOP is sufficient; there may not be a need to even use more than 2GB of VAS if you can decrease the quality of stuff until it fits. But TSOP's not the end-all, be-all solution. There are other, complementary solutions that also help. Such as this one.

2. Getting a 64-bit executable: This would require the source code, and potentially a lot of invasive changes, if the code has been written in a way that assumes a 32-bit environment. So depending on how the code was written, this could be very easy, or excruciatingly hard (it's especially hard if they use hand-coded x86 assembly). This WOULD BE the end-all, be-all solution, but nobody short of Ironclad can deliver it to us. And I think they have fully moved on from Sins by now.

3. Running it in an emulator like wine won't remove the VAS restrictions. On Linux, the wine program that runs 32-bit Windows executables is itself a 32-bit Linux executable. On Linux, we have the same problems you have with virtual address space. So the problem still lingers. In fact, it will probably be worse running under wine, because wine uses a lot of extra RAM by emulating all the Windows calls *on top of* Linux libraries like OpenGL and ALSA. If you build a 64-bit version of Wine, the native side will be 64-bit, but it can only run 64-bit Windows programs! So you can't have it both ways; it is impossible to run a 32-bit Windows program using a 64-bit version of Wine. Therefore the problem isn't addressed at all (no pun intended) by Wine. This comes despite the fact that I am a fan of and user of Wine; it's just the truth.

Yes, and your post is not exempt from this assessment.

Just to prove to you that LAA *does*, in fact, open up more address space, I built a very simple Microsoft C++ program that allocates memory until it can't allocate anymore, then it reports on what it did.

The way the heap works on Windows, you can only allocate -- within a *single object* -- a certain amount of data. This amount is determined by things like memory fragmentation, and the libraries that have been loaded into your process space. At first, this amount is really huge -- way more than you'd ever need in a single object, even for a program like Sins. The Sins image is composed of many independent objects that are relatively small, not one huge chunk of contiguous data; so this principle applies to Sins as well.

What my program does is it allocates as much data as it can in a buffer, then stores that buffer away, creates a new buffer, and fills that one up as much as it can. This process is repeated forever, or until the system says it can't allocate any data at all. Eventually, the system does in fact tell you it can't allocate any data at all; at this point, you know your process is REALLY out of memory. If you can't allocate a buffer with 1 byte, you're done.

Once it has completely filled up the process's virtual address space, it sums together all the sizes of the buffers it allocated, and prints them. It also prints a line each time a new buffer has been stored away. All of the buffers are stored cumulatively (at the same time), so the process is literally reserving all of that memory just like Sins would.

I then created two copies of the program. They differ ONLY in the fact that one of them, "prooflaa.exe", has the Large Address Aware flag set. The other one, "proof.exe", does not.

Try to run them on a system that either has the /3GB switch (Windows XP 32-bit), the IncreaseUserVA switch (Windows Vista / 7 32-bit -- see http://usa.autodesk.com/adsk/servlet/ps/dl/item?linkID=9240697&id=9583842&siteID=123112 ), or any 64-bit operating system (Windows XP / Vista / 7 64-bit).

What you'll find is that the end result of "prooflaa.exe" is approximately 1GB larger (in the case of a 32-bit OS) or 2GB larger (in the case of a 64-bit OS) than the end result of "proof.exe".

If Large Address Aware isn't having any effect, then explain the results? Clearly, since you are lecturing a software engineer on a highly technical topic, you'll have no problem pulling apart the source code I wrote, attached in the zip file. You can either open up the solution in Visual Studio 2010; or, you can just look at the source code in proof/proof/proof-2gb.c (sorry for the directory structure; Visual Studio sucks.) Pelaa isn't that complicated at only a few hundred lines, but the proof program is even simpler, at only 69 lines, including whitespace. This is the length of programs that most people write in their freshman year at college, so I'm sure this will be manageable for you if you are short on time.

To wrap this up: the source code and the executables are at http://tiyukquellmalz.org/pelaa/proof-laa.zip . To run both the executables and compare them against eachother, just run the laa.bat file. Or you can run them individually from the command prompt, or whatever you like. Again, for any trust issues I direct you to a capable virus scanner and/or the source code attached. And if you are wondering about the included "pelaa.exe", this is my program I talked about in the original post that (a) sets the LAA bit, or (b) tells you whether the LAA bit is set on a given program. I'm using the feature in (b) in the laa.bat script: to show you that prooflaa.exe is Large Address Aware, while proof.exe is not.

Both the proof programs and pelaa are open source; if you think these results must be mistaken, then look at the source and find the mistake.

Once you accept the reality of this, you will probably claim that there are 100 different ways to set the LAA bit and my program doesn't matter because it's redundant. Well, fine; but point to another program that will set all of your Sins executables to large address aware with a single double-click. No navigating the directory structure, no typing into the command prompt, no downloading Visual Studio. This is answered in my FAQ, in fact.

This program is really just begging to be bundled in peoples' mods. There are people out there who still haven't set their Sins executables to large address aware, because all of the existing solutions come off as "too advanced" for them. They want it stupidly easy. I understand and sympathize with that desire for simplicity. So here it is.

Just to put the nail in the coffin, I stumbled upon this great article, by a Microsoft employee in 2008, that basically says the exact same thing that I just said, but with pretty pictures, and his own executable.

http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

allquixotic, I do not like to get into fights or arguments, BUT as a person that HAS been programming since 1977 AND has seen the entire industry and the development of computers and programming environments over the same length of time.

from the WAY you are using the term VIRTUAL memory it looks like you actually mean APPLICATION DATA SPACE, NOT windows virtual memory( which to longterm programmers and technicians means the windows swap space on the hard drive), the LAA ONLY allows programs (if the code was written in a way that allows scalable address space) to have a LARGER APPLICATION DATA SPACE if particular flags ARE set in your windows configuration, BUT the default windows installation DOES NOT set those flags. so from this AND previous experience with LAA and other applications, is that the LAA flags in 32 bit windows DOES NOT help with system stability.

the next point is that you are seperating the application space and system space in the ram, and both ARE in the SAME 4GB ram space, with the main reason for the swap space to be to hold programs that are NOT being executed at this time, as the ram is used by 1 BIOS, 2 video card, 3 i/o locations, 4 kernel, 5 i/o drivers, 6 disk cache, AND 7 user programs, with the probability of being put into the swap area on the HDD being only for the user programs and AT MOST a reduction of the disk cache space as all the other items in memory ARE EITHER fixed (like the BIOS, video card & i/o locations) or ESSENTIAL and in continious use( like the kernel, i/o drivers and disk cache).

and before you start ranting at me, yes programs CAN be moved around in the address space, so any program that tries to access a particular BYTE of ram by physical ram address WILL get an illegal instruction error, so all programs should use data in their allocated data space which the operating system CAN alter the size of at the request of the progam within the limits that the operating system allows(btw this also goes back to the days of MSDOS 2.1 and T.S.R. drivers) which is how the ORGINAL windows 1,2 & 3 worked.

harpo

harpo, how does anything you said contradict what I wrote?

Using caps and big fonts doesn't seem to make your argument more valid. Here is your argument, summed up, if I interpreted it correctly:

(1) You don't like the way I'm using the terminology "virtual memory" and would rather that I call it "application data space". If you read my post, you'll find that I call the system-level virtual-to-physical mapping "virtual memory", and I call the "application data space" the "virtual address space". These are standard terminology in college textbooks in the 2000s. However, I consider a terminology war to be utterly pointless. If you want to call virtual memory the Super Duper Whingding Controller Of the Universe, go right ahead. The logical force of the argument is identical.

(2) You say that the flag that makes LAA work (which in Windows XP is the /3GB switch, and in Vista/7 the IncreaseUserVa switch) is not set by default. This is true. But, so what? Sins of a Solar Empire isn't installed by default, either, but you don't hear people complaining about that. If you're seriously concerned that people won't benefit from the LAA switch because they're running a 32-bit OS without the 3GB/IncreaseUserVa switch, I can modify pelaa-sins to automatically set these switches, and then tell the user that they should reboot to take advantage of it. Would that make you happy?

Furthermore, on 64-bit Windows, there's no need to set any flag in the operating system. Once you've set the LAA bit, your process can immediately address up to 4GB of RAM.

(3) You say that "if the code was written in a way that allows scalable address space" then LAA will work. What you really mean is "if the code was written in a way that properly respects the operating system's ability to use the full 32 bits of pointers for memory locations". To be frank, any applications written in the 2000s, at the least, should absolutely, positively not poke around inside pointers given to them by the operating system. The only case in which LAA won't work is if the application does poke inside pointers. Let me demonstrate.

So you obviously understand, as a fellow programmer, that you can represent a 32-bit pointer as 32 0s and 1s. I'm going to use Least Sign Bit notation, so that the last bit on the right is the highest bit.

Now, by default on older versions of Windows, Microsoft explicitly stated that they will only use the lowest 31 bits of the pointer for the address space (2GB), and the 32nd bit is basically "wasted" -- it has no purpose, it isn't read by the operating system. So people with a mind for squeezing an extra 4 bits of RAM savings out of their program said, "Cool! Let's store random boolean values in that 32nd bit!" So here is how it'd look like.

In the below  sequence, an "X" denotes "this bit is used by the operating system", and a "0" denotes "this bit is unused".

XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXX0

See that 0 on the end? That's where bad programmers tried to stuff their boolean values, rather than having another member in the struct that the pointer dereferences to.

The problem is, assuming 2GB is just an implementation detail. There is nothing in the C or C++ standards that entitles you to assume that you can mess around inside pointers. By messing around with pointers in this way, you are making your program non-portable. Specifically, it is not portable to another version of Windows that has the following arrangement for pointers:

XXXX XXXX XXXX XXXX  XXXX XXXX XXXX XXXX

That is, every single bit (even the highest bit) is significant information regarding the location of the pointer in VAS (virtual address space).

Fortunately, Sins of a Solar Empire is one application that does not exploit this non-portable behavior! So the good news is that Sins developers had the common sense not to make their code non-portable in an environment where Large Address Aware is in use. The bad news is that the developers did not have the common sense to enable /LARGEADDRESSAWARE when compiling the executables. Hence the need for pelaa-sins.

The claim that some programs, written by poor programmers trying to save a few bits in the early 90s, are incompatible with LAA, is completely moot in this case, because Sins of a Solar Empire specifically is not subject to this problem. Furthermore, every library Sins depends on also does not fall victim to this non-portable trap.

(4) You claim that LAA doesn't help with "system stability". I never claimed it did. If you don't have enough RAM to allow Sins to commit 4 GB, you're going to start swapping eventually, which can negatively affect system stability. Pushing Sins to use between 2.1 and 4 GB of committed pages is not for the faint of heart. In reality, you need about 6 - 8 GB of RAM (or more) to do this without having anything get swapped.

However, if you meant "the stability of Sins of a Solar Empire", then yes, LAA will help it, because memory allocations won't throw an out of memory exception (thus crashing the program) once 2GB have been allocated.

(5) Your last two paragraphs don't contradict what I wrote in any way. In fact, you seem to be agreeing with me. If you think you're disagreeing, then maybe you need to re-examine my earlier post (my first reply to Stant123).

I'm not used to seeing you getting this hot under the collar about a few terminology points, harpo. I am just stating the facts and using industry-standard terminology. I honestly don't mind if you want to use different terminology, and I do understand that Microsoft has used the term "virtual memory" in a different context in the past. The unfortunate fact is that the term "virtual memory" is plagued with a long past of misuses and misapplications. So when someone does use it, they ought to specify exactly what they mean, which I hope I've done sufficiently.

My eyes hurt. And I'm obviously stupid.

btw... does it work or not?

You won't know until you try

Like I said, it's not a universal antidote to crashing, but it may help you if you've got enough RAM and are fighting very large battles. Oh -- and if you've got a 64-bit operating system, it will help more than if you only have a 32-bit one.

At the end of the day, the truth is that it can't hurt. It can only help. Those with a 64-bit operating system and lots of RAM will benefit the most; those using sub-standard amounts of RAM on 32-bit OSes will benefit less.