Virtual Address Space Layouts

This section describes the components in the user and system address space, followed by the specific layouts on 32-bit and 64-bit systems. This information helps you to understand the limits on process and system virtual memory on both platforms.

Three main types of data are mapped into the virtual address space in Windows: per-process private code and data, sessionwide code and data, and systemwide code and data.

As explained in Chapter 1, each process has a private address space that cannot be accessed by other processes (unless they have permission to open the process for read or write access). Threads within the process can never access virtual addresses outside this private address space, unless they map to shared memory sections and/or use the cross-process memory functions that allow access to another process's address space. The information that describes the process virtual address space, called page tables, is described in the section on address translation. Page tables are marked as kernel-mode-only accessible pages so that user threads in a process cannot modify their own address space layout.

On systems with multiple sessions (Windows 2000 Server with Terminal Services installed, Windows XP, and Windows Server 2003), session space contains information global to each session. (For a description of sessions, see Chapter 2.) A session consists of the processes and other system objects (such as the window station, desktops, and windows) that represent a single user's workstation logon session. Each session has a session-specific paged pool area used by the kernel-mode portion of the Windows subsystem (Win32k.sys) to allocate session-private GUI data structures. In addition, each session has its own copy of the Windows subsystem process (Csrss.exe) and logon process (Winlogon.exe). The session manager process (Smss.exe) is responsible for creating new sessions, which includes loading a session-private copy of Win32k.sys, creating the session-private object manager namespace, and creating the session-specific instances of the Csrss and Winlogon processes.

To virtualize sessions, all sessionwide data structures are mapped into a region of system space called session space. When a process is created, this range of addresses is mapped to the pages appropriate to the session that the process belongs to. The size of the space for mapped views in session space can be configured by modifying registry values under HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management. (For 32-bit systems, these registry values are ignored if the system is booted /3GB.)

Finally, system space contains global operating system code and data structures visible to each process. System space consists of the following components:

• System code Contains the operating system image, HAL, and device drivers used to boot the system.

• System mapped views Used to map Win32k.sys, the loadable kernel-mode part of the Windows subsystem, as well as kernel-mode graphics drivers it uses. (See Chapter 2 for more information on Win32k.sys.)

• Hyperspace A special region used to map the process working set list and to temporarily map other physical pages for operations such as zeroing a page on the free list (when the zero list is empty and a zero page is needed), invalidating page table entries in other page tables (such as when a page is removed from the standby list), and process creation to set up a new process's address space.

• System working set list The working set list data structures that describe the system working set.*

• System cache Virtual address space used to map files open in the system cache. (See Chapter 11 for detailed information about the cache manager.)

• Paged pool Pageable system memory heap.

• System page table entries (PTEs) Pool of system PTEs used to map system pages such as I/O space, kernel stacks, and memory descriptor lists. You can see how many system PTEs are available by examining the value of the Memory: Free System Page Table Entries counter in the Performance tool.

• Nonpaged pool Nonpageable system memory heap, usually existing in two parts one in the lower end of system space and one in the upper end.

• Crash dump information Reserved to record information about the state of a system crash.

• HAL usage System memory reserved for HAL-specific structures.

Note

*Internally, the system working set is called the system cache working set. This term is misleading, however, because it includes not only the system cache but also the paged pool, pageable system code and data, and pageable driver code and data.

Now that we've described the basic components of the virtual address space in Windows, let's examine the specific layout on the x86, IA-64, and x64 platforms.

x86 User Address Space Layouts

By default, each user process on the 32-bit version of Windows can have up to a 2-GB private address space; the operating system takes the remaining 2 GB. Windows 2000 Advanced Server, Windows 2000 Datacenter Server, Windows XP Service Pack 2 and later, and Windows Server 2003 (all versions) support a boot-time option (the /3GB switch in Boot.ini) that allows 3-GB user address spaces. Windows XP and Windows Server 2003 support an additional switch (/USERVA), which allows specifying a user address space size between 2 GB and 3 GB. (The value is in number of megabytes.) These two possible address space layouts are shown in Figure 7-10.

Figure 7-10. x86 virtual address space layouts

The ability for a 32-bit process to grow beyond 2 GB was added as a short-term solution to accommodate the need for applications such as database servers to keep more data in memory than could be done with a 2-GB address space. The AWE functions described earlier in this chapter provide a much better solution to the need for accessing more data than can fit in the limited 2-GB (or 3-GB) process address space.

For a process to grow beyond 2 GB of address space, the image file must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header. Otherwise, Windows reserves the additional address space for that process so that the application won't see virtual addresses greater than 0x7FFFFFFF. (This is to avoid breaking applications that might break with such addresses.) You set this flag by specifying the linker flag /LARGEADDRESSAWARE when building the executable. This flag has no effect when running the application on a system with a 2-GB user address space. (If you boot with /3GB in versions of Windows Server that do not support the /3GB switch, system space is reduced to 1 GB, but user processes are still limited to 2 GB, even if the large-address-space aware flag is set on an image that is run.)

Several system images are marked as large address space aware so that they can take advantage of systems running with large process address spaces. These include:

• Lsass.exe the Local Security Authority Subsystem

• Inetinfo.exe Internet Information Server

• Chkdsk.exe the Check Disk utility

• Dllhst3g.exe a special version of Dllhost.exe (for COM+ applications)

Finally, because by default, memory allocations using VirtualAlloc start with low virtual addresses and grow higher, unless a process allocates a lot of virtual memory or it has a very fragmented virtual address space, it will never get back very high virtual addresses. Therefore, for testing purposes, you can force memory allocations to start from high addresses by adding a DWORD registry value, HKLM\System\CurrentControlSet\Control\Session Manager\ Memory Management\AllocationPreference, and setting it to 0x100000.

x86 System Address Space Layout

This section describes the detailed layout and contents of system space in 32-bit Windows. Figure 7-11 shows the overall structure on x86 systems with a 2-GB system space.

Figure 7-11. x86 system space layout (not proportional)

Table 7-8 lists the kernel variables that contain start and end addresses of various system space regions. Some of these regions are fixed; some are computed at system boot time on the basis of memory size and whether the system is running a client or server edition of Windows.

x86 Session Space

For systems with multiple sessions, the code and data unique to each session is mapped into system address space but shared by the processes in that session. Figure 7-12 shows the general layout of session space.

Figure 7-12. x86 session space layout (not proportional)

The sizes of the components of session space can be configured by adding registry values under the key HKLM\System\CurrentControlSet\Session Manager\Memory Management. These registry values along with the kernel variables that contain the actual sizes are listed in Table 7-9.

Table 7-9. Session Space Configuration Values

Registry Value Under	Kernel Variable
SessionImageSize	MmSessionImageSize
SessionViewSize	MmSessionViewSize
SessionPoolSize	MmSessionPoolSize

EXPERIMENT: Viewing Sessions

You can display which processes are members of which sessions by examining the session ID process performance counter. This can be viewed with Task Manager, Process Explorer, or the Performance Tool. Using the kernel debugger, you can list the active sessions with the !session command as follows:

lkd> !session Sessions on machine: 3 Valid Sessions: 0 1 2 Current Session 0

Then you can set the active session using the !session s command and display the address of the session data structures and the processes in that session with the !sprocess command:

lkd> !session -s 2 Sessions on machine: 3 Implicit process is now 8144f3a0 Using session 2 lkd> !sprocess Dumping Session 2 _MM_SESSION_SPACE f9d5a000 _MMSESSION        f9d5ad80 PROCESS 8144f3a0  SessionId: 2 Cid: 0d18 Peb: 7ffdf000 ParentCid:  0134     DirBase: 07ef7000 ObjectTable: e1855bb0 HandleCount: 71.     Image: csrss.exe PROCESS 813f6500 SessionId: 2 Cid: 0d04 Peb: 7ffdf000 ParentCid: 0134     DirBase: 0f3fd000 ObjectTable: e18f3008 HandleCount: 131.     Image: winlogon.exe

To view the details of the session, dump the MM_SESSION_SPACE structure using the dt command as follows:

lkd> dt  nt!_mm_session_space f9d4a000 nt!_MM_SESSION_SPACE    +0x000 GlobalVirtualAddress : 0xf9d4a000    +0x004 ReferenceCount      : 0x19    +0x008 u                :  __unnamed    +0x00c SessionId        : 0    +0x010 ProcessList      : _LIST_ENTRY[  0x817d92cc       -0x812de88c ]    +0x018 LastProcessSwappedOutTime : _LARGE_INTEGER0x0    +0x020 SessionPageDirectoryIndex : 0x6a29    +0x024 NonPagablePages  : 0xf    +0x028 CommittedPages   : 0x3f    +0x02c PagedPoolStart   : 0xbc000000    +0x030 PagedPoolEnd     : 0xbc3fffff    +0x034 PagedPoolBasePde : 0xc0300bc0    +0x038 Color            : 0xd6    +0x03c ProcessOutSwapCount : 5    +0x040 SessionPoolAllocationFailures : [4] 0    +0x050 ImageList        : _LIST_ENTRY  [ 0x81801fa8 - 0x816933a0 ]    +0x058 LocaleId         : 0x409    +0x05c AttachCount      : 0    +0x060 AttachEvent      : _KEVENT    +0x070 LastProcess      : (null)    +0x074 ProcessReferenceToSession : 25    +0x078 WsListEntry      : _LIST_ENTRY  [ 0x8056d058 - 0x8056d058 ]    +0x080 Lookaside        : [26]  _GENERAL_LOOKASIDE    +0xd80 Session          : _MMSESSION    +0xdc0 PagedPoolMutex   : _KGUARDED_MUTEX    +0xde0 PagedPoolInfo    : _MM_PAGED_POOL_INFO    +0xe00 Vm               : _MMSUPPORT    +0xe60 Wsle             : 0xbf400038    +0xe64 Win32KDriverUnload : 0xbf9169ab  +ffffffffbf9169ab    +0xe68 PagedPool        : _POOL_DESCRIPTOR    +0x1e98 PageTables       : 0x816e7008    +0x1e9c ImageLoadingCount : 0

EXPERIMENT: Viewing Session Space Utilization

You can view session space memory utilization with the !vm 4 command in the kernel debugger. For example, the following output was taken from a 32-bit Windows Server 2003 Enterprise Edition system with two active sessions:

lkd> !vm  4 . . Terminal Server Memory Usage By Session: Session  Paged Pool  Maximum  is  4096K Session  View Space  Maximum  is  49152K Session  ID 0 @ f9d4a000: Paged Pool Usage:             0K Commit Usage:               252K Session  ID 1 @ f9d4c000: Paged Pool Usage:             0K Commit Usage:               172K

Executing the same command on an x64 Windows Server 2003 Enterprise Edition system with two active sessions shows the much larger session space regions:

Terminal Server Memory Usage By Session: Session Paged Pool Maximum is 65536K Session View Space Maximum is 106496K Session ID 0 @ fffffadfe4bb9000: Paged Pool Usage:           0K Commit Usage:            4608K Session ID 1 @ fffffadfe4c5f000: Paged Pool Usage:           0K Commit Usage:             584K

System Page Table Entries

System page table entries are used to dynamically map system pages such as I/O space, kernel stacks, and memory descriptor lists. System PTEs aren't an infinite resource. For example, Windows 2000 can describe only 660 MB of variable system virtual address space (440 MB of which could be contiguous). With the 32-bit edition of Windows XP and Windows Server 2003, the number of available system page table entries (PTEs) has increased such that the system can describe 1.3 GB of system virtual address space, 960 MB of which can be contiguous. On 64-bit Windows, system PTEs can describe up to 128 GB of contiguous virtual address space.

You can see how many system PTEs are available by examining the value of the Memory: Free System Page Table Entries counter in the Performance tool. By default, Windows calculates at boot time how many system PTEs to allocate based on the memory size. You can override this calculation by setting the registry value HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SystemPages to the number of PTEs you want. (This might be needed to support, for example, a device that requires a large number of system PTEs, such as a video card with 512 MB of video memory that has to be mapped all at one time.) If the value is 0xFFFFFFFF, the maximum number of system PTEs will be reserved.

64-Bit Address Space Layouts

The theoretical 64-bit virtual address space is 16 exabytes (18,446,744,073,709,551,616 bytes, or approximately 17.2 billion gigabytes). Unlike on the x86 where the default address space is divided in two parts (half for a process and half for the system), the 64-bit address is divided into a number of different sized regions whose components match conceptually the portions of user, system, and session space. The various sizes of these regions, listed in Table 7-10, represent current implementation limits that could easily be extended in future releases. (The sizes on the 32-bit x86 platform are included for comparison purposes.) Clearly, 64-bits provides a tremendous leap in terms of address space sizes.

Table 7-10. The 64-Bit Address Space Sizes

Region	IA-64	x64	x86
Process Address Space	7152 GB	8192 GB	2 to 3 GB
System PTE Space	128 GB	128 GB	1.2 GB
System Cache	128 GB	128 GB	960 MB
Paged Pool	128 GB	128 GB	470 to 650 MB
Nonpaged Pool	128GB	128 GB	256 MB

The detailed IA-64 and x64 address space layouts vary slightly. The IA-64 address space layout is shown in Figure 7-13, and the x64 address space layout is shown in Figure 7-14.

Figure 7-13. IA-64 address space layout

Figure 7-14. x64 address space layout