RAM, Virtual Memory, PageFile and all that stuff
Bruce Sanderson's General Windows Information
RAM, Virtual Memory, PageFile and all that stuff
This article explains some aspects of the Virtual Memory implementation in Windows 2000, XP, 2003 Server etc. It is intended for a general audience that has an interest in the relationship between RAM (Random Access Memory), Address Spaces, Virtual Memory and pagefile in those systems. An abbreviated version of this article is available at http://support.microsoft.com/?kbid=555223.
The article is arranged with basic information covering the scope of the article in 1. Basic Concepts with more details in 2. Additional Detail. This should enable the casual reader to get basic information without getting buried in detail, but those interested in a greater understanding can read the detailed material further down in the article.
This article is mainly concerned with 32 bit Windows versions, but most of the discussion is equally applicable to 64 bit versions. See 3. 64 Bit Processors and Address Spaces for some information specifically about 64 bit versions.
1.1. Virtual Memory
In most modern operating systems, including Windows, application programs and many system processes always reference memory using virtual memory addresses which are automatically translated to real (RAM) addresses by the hardware. Only core parts of the operating system kernel bypass this address translation and use real memory addresses directly. The virtual memory management component of the operating system maintains the tables used by the hardware to provide the mapping of virtual addresses into real addresses. A small amount of RAM is used by those parts of the operating system that can not be paged out (for example, the virtual memory manager) and is thus not available for assignment to virtual memory.
The key point here is that Virtual Memory is always in use, even when the memory required by all running processes does not exceed the amount of RAM installed on the system.
See 2.1 Virtual Memory for more information.
Any process (e.g. application executable) running under 32 bit Windows versions gets a set of virtual memory addresses (a Virtual Address Space) going from 0 to 4,294,967,295 (2*32-1 = 4 GB), no matter how much RAM is actually installed on the computer. Actually, this is essentially the same for all operating systems running on 32 bit hardware that implement Virtual Memory.
In the normal, default 32 bit Windows OS configuration, 2 GB of this virtual address space are allocated to the process’ private use and the other 2 GB are allocated to shared and operating system use.
Only that portion of the address space that is actually referenced (used) by a process or the system is associated with a page frame in real memory, or in the pagefile.
See 2.2 Address Spaces and Processes for more information, about features to make more virtual memory available to processes. For information about using more than 4 GB of installed RAM with 32 bit Windows, see 2.4 Physical Address Extension (PAE).
The only way to actually increase the size of the virtual address space beyond 4 GB is by using 64 bit hardware with an operating system and application built for that hardware. See 3. 64 Bit Processors and Address Spaces for additional information about 64 bit architectures.
RAM is a limited resource, whereas virtual memory is, to a large extent, unlimited in that there can be a large number of processes each with its own 4 GB virtual address space, of which 2 GB is private to the process (32 bit Windows). When the memory in use (that is, has been reference) by all the existing processes exceeds the amount of RAM available, the operating system will move pages (4 KB pieces) of one or more virtual address spaces to the computer’s hard disk, thus freeing some RAM for other uses. In Windows systems, these “paged out” pages are stored in one or more files with the name pagefile.sys. There can be one such file in each disk partition.
A frequently asked question is “how big should I make the pagefile”? There is no single answer to this question, because it depends on the workload on the computer, how much RAM there is and how much virtual memory space that workload requires.
With operating systems like Windows XP on older computers with a small amount of RAM it is not uncommon for power users to have many windows open (and thus processes running) and to exhaust the available RAM. In this situation, the goal is provide sufficient pagefile space to avoid running out of available virtual memory, which could cause application failures or at least, significant inconvenience.
There is quite a variety of ideas on this subject, but here’s a place to start for computers (typically older ones) that don't have much RAM:
Most newer computers will have more than I GB of RAM and, in most cases, there is sufficient RAM available that having more virtual memory than the available RAM is a relatively rare occurrence. There is little point in allocating a large minimum size pagefile, since it typically won’t be used. However, having a relatively large maximum will avoid “out of virtual memory” conditions without dedicating a big chunk of disk space for that rare situation.
A common objective in configuring a server is to ensure sufficient RAM so that paging doesn’t happen. Providing a large maximum pagefile is insurance against “out of virtual memory” situations that could cause service interruptions. Servers running a few applications are typically configured with enough RAM that paging is rare. A possible exception are servers used for Terminal Services, where it would be common for some users to be relative inactive for long periods of time and paging may occur routinely without a significant performance penalty.
There are several advantages to Virtual Memory implementations with per process virtual address spaces, some of which are:
A process is essentially, the combination of:
The only way to increase the size of the virtual address space for a process beyond 4 GB is to use 64 bit hardware with a 64 bit version of the operating system and application built for the 64 bit instruction set.
The nub of it is, that no matter how much physical RAM is in the computer, the amount of memory available in the process’ private part of the virtual address space in 32 bit Windows implementations is limited to:
With 64 bit operating systems, this 4 GB limit on address space is essentially removed - see 3. 64 Bit Processors and Address Spaces.
A common recommendation is to make the page file 1.5 times the size of the installed RAM. This recommendation makes sense only for computers with small amounts of RAM (256 MB or less). For example, there is usually not much point in allocating a page file that is 96 GB if the computer has 64 GB of RAM. The objective in such RAM rich systems is to avoid using the pagefile at all by providing sufficient RAM that all virtual memory can be in RAM all the time. If the virtual memory in use exceeds the amount of installed RAM, performance will suffer and having a larger pagefile will not help this situation.
Windows only assigns space in the pagefile when a virtual memory page must be paged out. Thus the total available virtual memory is approximately the sum of installed RAM and the size of the pagefile(s). See "Virtual Memory" at http://support.microsoft.com/?kbid=294418.
An argument sometimes made is that the minimum pagefile has to be at least the size of the RAM so that a full memory dump can be taken when the operating system crashes. However, this is not a very powerful argument because:
The operating system uses a small amount of space in the pagefile even when RAM is not exhausted, but in general terms, this is not particularly important. The main thing is that space in the page file will be allocated to virtual memory pages for which no corresponding RAM page is available.
If the minimum and maximum pagefile sizes are not the same, the pagefile will initially be allocated to the minimum size, then expanded as more space is needed, up to the maximum. On most systems, this will mean that the pagefile will be fragmented, possibly in widely separated parts of the disk. If there is a significant amount of paging activity, this can lead to unnecessary performance degradation. Setting the minimum and maximum the same size may still result in a fragmented pagefile, but does reduce the likelihood of serious fragmentation. If the system is paging heavily and the pagefile is heavily fragmented, consider moving the pagefile to a newly formatted partition on a separate disk.
The best way to size the pagefile is to monitor how much of the pagefile is in use and the system paging activity. If very little of the pagefile is ever used, there is not much point in having a large minimum size. Conversely if the pagefile is typically 80% or more used, increasing the minimum size may be advantageous. See 2.8. Monitoring RAM and Virtual Memory usage for more information about monitoring memory and pagefile use.
For best performance when there actually is paging activity, the pagefile should be on a disk that is otherwise not used or that has very little activity. If possible, the page file should be created on a freshly formatted disk partition (or one that has been de-fragmented). This will usually ensure that the pagefile is contiguous on disk, thus minimizing head motion and increasing performance.
However, if there is significant paging activity, the best way to improve performance is to add more RAM. This will be far more effective in improving performance than any optimization of the pagefile location or size.
PAE is a feature of Intel processors (Pentium Pro or later and others that implement the Intel architecture, such as those from AMD) so that 32 bit operating systems can address physical memory (RAM) above 4 GB. To support this, the processor, supporting chip sets and mother boards provide up to 36 physical address bits. Using the processor architecture defined registers and tables, 32 bit virtual addresses can be mapped into RAM pages above the 4 GB boundary. This hardware feature is supported by Windows NT, 2000, XP, 2003 and later. This feature is activated by using the /PAE switch in the boot.ini; for Vista and Server 2008 using bcdedit /set pae forcenable. PAE may be enabled automatically (without the /PAE boot setting) if the processor supports hardware DEP (Data Execution Prevention - see http://www.microsoft.com/whdc/system/platform/server/PAE/PAEdrv.mspx for more information).
Using this feature does not change the size of the virtual address space used by processes, but does allow the virtual memory for more processes to be resident in RAM, thus reducing paging in large system environments.
You may see statements that the PAE feature slows down the processor and thus should not be used, for example, on a Terminal Server. While some Intel documentation does say that the processor performance will be degraded if PAE is enabled, it doesn’t say by how much and this is most likely very small amount. Unless the bottleneck in your system is CPU time, the fact that the processor runs a bit slower (e.g. equivalent of 2.0 MHz instead of 2.1 MHz) will be hardly noticeable or measurable (by the way, having multiple processors in a computer also degrades processor performance – but two processors does allow the system to handle a larger load, just not two times as much). The slowest process in almost any computer is disk access. So if your computer is doing a lot of paging and to use more RAM you need to enable the PAE feature the few percentage point slowdown in the CPU will most likely be more than offset by the reduction in paging activity due to the additional RAM. The only way to tell for sure is to do the measurements – e.g. monitor CPU time with Performance Monitor before and after enabling PAE with the same load (e.g. number of users).
With some hardware, even if PAE is enabled, 32 bit operating systems will not be able to use more than ~ 3GB of RAM because of the hardware architecture. Some of the physical address space is dedicated to some pieces of hardware, such video adapters. For additional details, see for example http://support.microsoft.com/kb/929605.
The maximum amount of physical RAM that is supported varies by 32 bit Windows version and edition. The list below is taken from the articles at:
For additional information on PAE and other large memory features see:
To accommodate those applications (e.g. mail servers, database management systems) that can effectively use private virtual memory larger than 2 GB, Windows NT 4, 2000 etc. provide the Application Memory Tuning feature (renamed to 4 GB RAM Tuning in Windows Server 2003 documentation) that moves the boundary between the private part of the 4 GB virtual address space from 2 GB to 1 GB so that the private part of the address space is 3 GB. This feature is activated by using the /3GB switch in the boot.ini file.
Note that if the /3GB switch is used, the maximum RAM addressable by any Windows version is 16 GB (see Large Memory Support Is Available in Windows 2000 and Windows Server 2003 http://support.microsoft.com/default.aspx?scid=kb;en-us;283037).
More information about this feature is available in the article Memory Support and Windows Operating Systems (http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx). Here are some of the details from that article:
2.5.1. The /3GB switch is available on the 32 bit version of these operating systems:
2.5.2. Application implications (a quote from http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx):
“No APIs are required to support application memory tuning. However, it would be ineffective to automatically provide every application with a 3-GB address space.
Executables that can use the 3-GB address space are required to have the bit IMAGE_FILE_LARGE_ADDRESS_AWARE set in their image header. If you are the developer of the executable, you can specify a linker flag (/LARGEADDRESSAWARE).
To set this bit, you must use Microsoft Visual Studio Version 6.0 or later and the Editbin.exe utility, which has the ability to modify the image header (/LARGEADDRESSAWARE) flag. For more information on setting this flag, see the Microsoft Visual Studio documentation.”
2.5.3. The /3GB switch does not apply to 64 bit versions of windows. All 64 bit Windows versions will provide 4 GB of private address space to 32 bit processes (applications) if the IMAGE_FILE_LARGE_ADDRESS_AWARE bit is set in their image header (see previous section). See http://msdn.microsoft.com/en-us/library/aa366778.aspx and http://support.microsoft.com/?kbid=294418 for additional information.
This feature (also known as AWE) is a set of functions (APIs) that an application can use to request the operating system to map pages of RAM into its 4 GB virtual address space. Using this feature, an application can thus use more RAM than the usual 2 GB (or 3 GB if the /3GB switch is used) available in the private part of the virtual memory. This can be useful for applications that can benefit from more memory than is normally available. There are two important things about this feature to understand:
More information about this feature is available in the article Address Windowing Extensions (http://msdn2.microsoft.com/en-us/library/Aa366527.aspx).
On any computer system, as load (number of users, amount of work being done) increases, performance (how long it takes to do each task) will decrease, but in a non linear fashion. Any increase in load (demand) beyond a certain point will result in a dramatic decrease in performance. This means that some resource is in critically short supply and has become a bottleneck.
At some point, the resource in critical short supply can not be increased. This means an architectural limit has been reached. Some commonly reported architectural limits in 32 bit Windows include:
1. 2 GB of shared virtual address space for the system
2. 2 GB of private virtual address space per process
3. 660 MB System PTE storage
4. 470 MB paged pool storage
5. 256 MB non-paged pool storage
6. 1 GB System cache
7. 16,000 GB pagefile size
The above applies to Windows 2003 Server specifically (from http://support.microsoft.com/?kbid=294418), but also apply to Windows XP and Windows 2000.
Commonly found and quoted statements such as:
with a Terminal Server, the 2 GB of shared address space will be completely used before 4 GB of RAM is used
may be true in some situations, but you need to monitor your system to know whether they apply to your particular system or not. In some cases, these statements are conclusions from specific Windows NT 4.0 or Windows 2000 environments and don't necessarily apply to Windows Server 2003 or 2008. Significant changes were made to Windows Server 2003 and 2008 to reduce the likelihood that these architectural limits will in fact be reached in practice. For example, some processes that were in the kernel have been moved to non kernel processes to reduce the amount of memory used in the shared virtual address space.
Performance Monitor (Start, Administrative Tools, Performance) is the principle tool for monitoring system performance and identifying what the bottleneck really is. In Vista and Server 2008, the Resource Monitor (can be launched from the Performance tab of Task Manager) provides additional load and performance information, but that is not discussed here. Here's a summary of some important counters and what they tell you.
Memory, Committed Bytes - this is a measure of the demand for virtual memory
This counter shows how many bytes have been allocated by processes and to which the operating system has committed a RAM page frame or a page slot in the pagefile (perhaps both). As Committed Bytes grows above the amount of available RAM, paging will increase and the amount of the pagefile in use will also increase. At some point, paging activity will start to significantly impact perceived performance.
Memory, Pages/Sec - this is one of the most misunderstood measures.
A high value for this counter does not necessarily imply that your performance bottleneck is shortage of RAM. The operating system uses the paging system for purposes other than swapping pages due to memory over commitment.
This counter counts all page activities, which includes:
See http://support.microsoft.com/?kbid=139609 for more information about Memory: Pages/sec.
Memory: Pages Input/sec- this shows how many virtual memory pages were read from the pagefile into a free RAM page frame each second.
This is the best counter to monitor if you suspect that paging is your performance bottleneck. Even if Memory, Committed Bytes is greater than the installed RAM, if Pages Input/sec is low or zero most of the time, there is not a significant performance problem from not enough RAM. In most cases, a page in operation means that some process has been suspended because the virtual memory it referenced is not in RAM and had to be reloaded from the pagefile. Some processes may be idle for a long time (e.g. a Terminal Server user is away from their workstation for some reason). When such an idle process becomes active again, the operating system will reload some paged out pages in anticipation that they will be required. This pre-loading of pages reduces processing delays and improves performance. On a workstation or Terminal Server, users may notice a degradation in performance if their processes experience a significant number of page in operations.
Memory, Pages Output/Sec - this shows how many virtual memory pages were written to the pagefile to free RAM page frames for other purposes each second.
This is an important counter to monitor if you suspect that paging is your performance bottleneck. Even if Memory, Committed Bytes is greater than the installed RAM, if Pages Output/sec is low or zero most of the time, there is not a significant performance problem from not enough RAM. If RAM is in short supply, the operating system will pre-emptively steal pages from idle processes so that there are always empty RAM page frames available for assignment to running processes, thus minimizing delays for the active processes.
Memory, Cache Bytes
Memory, Pool Nonpaged Bytes
Memory, Pool Paged Bytes
Memory, System Code Total Bytes
Memory, System Driver Total Bytes
The sum of these counters is a measure of how much of the 2GB of the shared part of the 4 GB virtual address space is actually in use. Use these to determine if your system is reaching one of the architectural limits discussed above.
Memory, Available MBytes - this measures how much RAM is available to satisfy demands for virtual memory (either new allocations, or for restoring a page from the pagefile).
When RAM is in short supply (e.g. Memory, Committed Bytes is greater than installed RAM), the operating system will attempt to keep a certain fraction of installed RAM available for immediate use by copying virtual memory pages that are not in active use to the pagefile. For this reason, this counter will not go to zero and is not necessarily a good indication of whether your system is short of RAM. This counter does measure the demand for RAM, only how much the OS has kept free for immediate use.
Process, Working Set, _Total - this is a measure of the amount of virtual memory in "active" use
This counter shows how much RAM is required so that the actively used virtual memory for all processes is in RAM. This is always a multiple of 4,096, which is the page size used in Windows. As demand for virtual memory increases above the available RAM, the operating system will adjust how much of a process's virtual memory is in its Working Set to optimize the use of available RAM and minimize paging.
Paging File, %pagefile in use - this is a measure of how much of the pagefile is actually being used
This is the counter to use to determine if the pagefile is an appropriate size. If this counter gets to 100, the pagefile is completely full and things will stop working. Depending on the volatility of your workload, you probably want the pagefile large enough so that it is normally no more than 50 - 75% used. If a lot of the pagefile is in use, having more than one on different physical disks, may improve performance.
As mentioned in 1.2 Address Spaces and Process, 64 bit processor architectures provide virtual and physical address spaces greater than 4 GB, which can be advantageous for some applications. 64 bit architectures can provide other advantages to applications that are not directly related to the amount of addressable memory, but those are not discussed here.
It is important to realize that a 64 bit architecture does not necessarily mean a 64 bit virtual or physical (RAM) address space is available. Processor, mother board, chip set and operating system design may well limit the available virtual or real addresses to less than 64 bits.
The size of the address spaces provided by these architectures appears, at this juncture, to be more than one could imagine using. However, history has shown that our ability to predict address space requirements for future systems has been pretty poor. For example:
Maximum RAM supported by 64 bit editions of Windows:
http://www.microsoft.com/downloads/details.aspx?FamilyID=bbc16ebf-4823-4a12-afe1-5b40b2ad3725&DisplayLang=en (Windows Vista Product Guide - Windows Vista Editions Feature Table)
Architectural limits (see 2.7. Performance, Architectural Limits and RAM) for 64 bit Windows memory are:
1. 8 TB of shared virtual address space for the system
2. 8 TB of private virtual address space per process
3. 128 GB System PTE storage
4. 128 GB paged pool storage
5. 128 GB non-paged pool storage
6. 1 TB System cache
7. 256 TB pagefile size
Here’s some information from various sources about address space implementations in various 64 bit processor architectures from Intel and AMD.
Not all processors actually have (implement) the same number of physical and virtual address bits. However, all the relatively new processors support larger physical address spaces than are actually supported by any version of Windows, so the actual number of physical address bits implemented is somewhat irrelevant.
For virtual addresses, the operating system must ensure that it does not attempt to allocate virtual memory addresses using un-implemented virtual address bits because there will be application or system failures if it does. The processors provide a way for the operating system to determine the number of virtual address bits and thus the maximum useable virtual address actually implemented (e.g. on Itanium processors, the PAL_VM_SUMMARY "procedure" - see page 2-450 in http://download.intel.com/design/Itanium/manuals/24531805.pdf ).
Itanium (see http://www.intel.com/design/itanium/itanium/itaniumprodbrief.pdf and http://download.intel.com/design/Itanium/manuals/24531805.pdf section 4.3.2)
Itanium 2 (see http://download.intel.com/design/Itanium/manuals/24531805.pdf)
Intel Pentium, Xeon, Core 64 bit (see http://download.intel.com/design/processor/manuals/253665.pdf)
Athlon, Opteron, Sempron and Turion:
Last updated 19 Sep 2010