Introduction to the Memory Manager

 < Day Day Up > 

By default, the virtual size of a process on 32-bit Windows is 2 GB. If the image is marked specially as large address space aware, and the system is booted with a special switch (described later in this chapter), a 32-bit process can grow to 3 GB on 32-bit Windows and up to 4 GB on 64-bit Windows. The process virtual address space size on 64-bit Windows is 7152 GB on IA-64 systems and 8192 GB on x64 systems. (This value could be increased in future releases.)

As you saw in Chapter 2 (specifically in Table 2-4), the maximum amount of physical memory supported by Windows ranges from 2 GB to 1024 GB, depending on which version and edition of Windows you are running. Because the virtual address space might be larger or smaller than the physical memory on the machine, the memory manager has two primary tasks:

  • Translating, or mapping, a process's virtual address space into physical memory so that when a thread running in the context of that process reads or writes to the virtual address space, the correct physical address is referenced. (The subset of a process's virtual address space that is physically resident is called the working set. Working sets are described in more detail later in this chapter.)

  • Paging some of the contents of memory to disk when it becomes overcommitted that is, when running threads or system code try to use more physical memory than is currently available and bringing the contents back into physical memory when needed.

In addition to providing virtual memory management, the memory manager provides a core set of services on which the various Windows environment subsystems are built. These services include memory mapped files (internally called section objects), copy-on-write memory, and support for applications using large, sparse address spaces. In addition, the memory manager provides a way for a process to allocate and use larger amounts of physical memory than can be mapped into the process virtual address space (for example, on 32-bit systems with more than 4 GB of physical memory). This is explained in the section "Address Windowing Extensions" later in this chapter.

Memory Manager Components

The memory manager is part of the Windows executive and therefore exists in the file Ntoskrnl.exe. No parts of the memory manager exist in the HAL. The memory manager consists of the following components:

  • A set of executive system services for allocating, deallocating, and managing virtual memory, most of which are exposed through the Windows API or kernel-mode device driver interfaces

  • A translation-not-valid and access fault trap handler for resolving hardware-detected memory management exceptions and making virtual pages resident on behalf of a process

  • Several key components that run in the context of six different kernel-mode system threads:

    • The working set manager (priority 16), which the balance set manager (a system thread that the kernel creates) calls once per second as well as when free memory falls below a certain threshold, drives the overall memory management policies, such as working set trimming, aging, and modified page writing.

    • The process/stack swapper (priority 23) performs both process and kernel thread stack inswapping and outswapping. The balance set manager and the threadscheduling code in the kernel awaken this thread when an inswap or outswap operation needs to take place.

    • The modified page writer (priority 17) writes dirty pages on the modified list back to the appropriate paging files. This thread is awakened when the size of the modified list needs to be reduced. (See the section "Modified Page Writer" to find out how you can change this default value.)

    • The mapped page writer (priority 17) writes dirty pages in mapped files to disk. It is awakened when the size of the modified list needs to be reduced or if pages for mapped files have been on the modified list for more than 5 minutes. This second modified page writer thread is necessary because it can generate page faults that result in requests for free pages. If there were no free pages and there was only one modified page writer thread, the system could deadlock waiting for free pages.

    • The dereference segment thread (priority 18) is responsible for cache reduction as well as page file growth and shrinkage. (For example, if there is no virtual address space for paged pool growth, this thread trims the page cache so that the paged pool used to anchor it can be freed for reuse.)

    • The zero page thread (priority 0) zeroes out pages on the free list so that a cache of zero pages is available to satisfy future demand-zero page faults. (Memory zeroing in some cases is done by a faster function called MiZeroInParallel. See the note in the section "Page List Dynamics.")

Each of these components is covered in more detail later in the chapter.

Internal Synchronization

Like all other components of the Windows executive, the memory manager is fully reentrant and supports simultaneous execution on multiprocessor systems that is, it allows two threads to acquire resources in such a way that they don't corrupt each other's data. To accomplish the goal of being fully reentrant, the memory manager uses several different internal synchronization mechanisms to control access to its own internal data structures, such as spinlocks and executive resources. (Synchronization objects are discussed in Chapter 3.)

Systemwide resources to which the memory manager must synchronize access include the page frame number (PFN) database (controlled by a spinlock), section objects and the system working set (controlled by pushlocks), and page file creation (controlled by a mutex). In Windows XP and Windows Server 2003, a number of these locks have been either removed completely or optimized, resulting in much less contention. For example, in Windows 2000, spinlocks were used to synchronize changes to system address space and memory commitment; however to improve scalability, these spinlocks have been removed as of Windows XP. Per-process memory management data structures that require synchronization include the working set lock (held while changes are being made to the working set list) and the address space lock (held whenever the address space is being changed). Working set synchronization in Windows 2000 was implemented as a mutex; in Windows XP and later, a pushlock is used, improving parallelism and scalability, because pushlocks support both shared and exclusive access.

Other operations that no longer involve acquiring locks include charging nonpaged and paged pool quotas, charging commitment of pages, and allocating and mapping physical memory allocated through the address windowing extensions (AWE) functions. Also, the lock that synchronizes access to the structures that describe physical memory (the PFN database) is now acquired less and when acquired, held for less time. These changes translate into greater parallelism and scalability on multiprocessor systems because the number of times the memory manager might have to block while another CPU is making a change to a global structure has been reduced or eliminated.

Configuring the Memory Manager

Like most of Windows, the memory manager attempts to automatically provide optimal system performance for varying workloads on systems of varying sizes and types. While there are a limited number of registry values you can add and/or modify under the key HKLM\ SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management to override some of these default performance calculations, in general, the memory manager's default computations will be sufficient for the majority of workloads.

Many of the thresholds and limits that control memory manager policy decisions are computed at system boot time on the basis of memory size and product type. (Windows 2000 Professional and Windows XP Professional and Home editions are optimized for desktop interactive use, and Windows Server systems are optimized for running server applications.) These values are stored in various kernel variables and later used by the memory manager. To find some of these, search for global variables in Ntoskrnl.exe that have names beginning with Mm that contain the word "maximum" or "minimum." (See the experiment "Peering into Undocumented Interfaces" in Chapter 2.)

Warning

Although you'll find references to some of these variables, you shouldn't change them. Windows has been tested to operate properly with the current possible permutations of these values that can be computed. Changing the value of these kernel variables on a running system can result in unpredictable system behavior, including system hangs or even crashes.


Examining Memory Usage

The Memory and Process performance counter objects provide access to most of the details about system and process memory utilization. Throughout the chapter, we'll include references to specific performance counters that contain information related to the component being described.

Besides the Performance tool, a number of tools in the Windows Support Tools and Windows resource kits display different subsets of memory usage information. We've included relevant examples and experiments throughout the chapter. One word of caution, however different utilities use varying and sometimes inconsistent or confusing names when displaying memory information. The following experiment illustrates this point. (We'll explain the terms used in this example in subsequent sections.)

EXPERIMENT: Viewing System Memory Information

The Performance tab in the Windows Task Manager, shown in the following screen shot from Windows XP, displays basic system memory information. This information is a subset of the detailed memory information available through the performance counters.



Both Pmon.exe (in the Windows Support Tools) and Pstat.exe (in the Platform SDK) display system and process memory information. The annotations in the following output from Pstat explain the information reported. (For an explanation of the commit total and limit, see Table 7-15.)

Table 7-15. Committed Memory and Page File Performance Counters

Performance Counter

Description

Memory: Committed Bytes

Number of bytes of virtual (not reserved) memory that has been committed. This number doesn't necessarily represent page file usage because it includes private committed pages in physical memory that have never been paged out. Rather, it represents the amount of page file space that would be used if the process was completely made nonresident.

Memory: Commit Limit

Number of bytes of virtual memory that can be committed without having to extend the paging files; if the paging files can be extended, this limit is soft.

Paging File: %

Usage Percentage of the paging file committed.

Paging File: %

Usage Peak Highest percentage of the paging file committed.




To see the specific usage of paged and nonpaged pool, use the Poolmon utility, described in the "Monitoring Pool Usage" section.

Finally, the !vm command in the kernel debugger shows the basic memory management information available through the memory-related performance counters. This command can be useful if you're looking at a crash dump or hung system. Here's an example of its output:

kd> !vm *** Virtual Memory Usage ***         Physical Memory: 32620 ( 130480 Kb)         Page File: \??\C:\pagefile.sys            Current:    204800Kb Free Space:    101052Kb            Minimum:    204800Kb Maximum:       204800Kb         Available Pages:     3604   (   14416 Kb)         ResAvail Pages:     24004   (   96016 Kb)         Modified Pages:       768   (    3072 Kb)         NonPagedPool Usage:  1436   (    5744 Kb)         NonPagedPool Max:   12940   (   51760 Kb)         PagedPool 0 Usage:   6817   (   27268 Kb)         PagedPool 1 Usage:    982   (    3928 Kb)         PagedPool 2 Usage:    984   (    3936 Kb)         PagedPool Usage:     8783   (   35132 Kb)         PagedPool Maximum:  26624   (  106496 Kb)         Shared Commit:       1361   (    5444 Kb)         Special Pool:           0   (       0 Kb)         Free System PTEs:  189291   (  757164 Kb)         Shared Process:      3165   (   12660 Kb)         PagedPool Commit:    8783   (   35132 Kb)         Driver Commit:       1098   (    4392 Kb)         Committed pages:    45113   (  180452 Kb)         Commit limit:       79556   (  318224 Kb)         Total Private:      30536   (  122144 Kb)             IEXPLORE.EXE       3028 (   12112 Kb)             svchost.exe        2128 (    8512 Kb)             WINWORD.EXE        1971 (    7884 Kb)             POWERPNT.EXE       1905 (    7620 Kb)             Acrobat.exe        1761 (    7044 Kb)             winlogon.exe       1361 (    5444 Kb)             explorer.exe       1300 (    5200 Kb)             livekd.exe         1015 (    4060 Kb)             hh.exe              960 (    3840 Kb)


EXPERIMENT: Accounting for Physical Memory Use

By combining information available from performance counters with output from kernel debugger commands, you can come close to accounting for physical memory usage on a machine running Windows. To examine the memory usage information available through performance counters, run the Performance tool and add the counters to view the following information. (You'll see the results more easily if you change the vertical scale maximum on the graph to 1000.)

  • Total process working set size To view this information, select the Process performance object and the Working Set counter for the _Total process instance. This number will be larger than the actual total process memory utilization because shared pages are counted in each process working set. To get a more accurate picture of process memory utilization, subtract free memory (available bytes), operating system memory used (nonpaged pool, resident paged pool, and resident operating system and driver code), and the size of the modified list from the total physical memory on the machine. What you're left with is the memory being used by processes. Comparing this value against the total process working set size as reported by the Performance tool gives you some indication of the amount of sharing occurring between processes. Although examining process physical memory usage is interesting, of more concern is the private committed virtual memory usage by processes, because memory leaks show up as an increasing private virtual size, not an increasing working set size. At some point, the memory manager will stop the process from growing in physical size, though it can continue to grow in its virtual size until the systemwide commit limit the maximum amount of private committed virtual memory available on the system is reached (or less if the process is a member of a job with a job-wide or process virtual memory limit). For more information, see the section "Page Files."

  • Total system working set size To view this information, select the Memory processor object and the Cache Bytes counter. As explained in the section "System Working Set," the total system working set size includes more than just the cache size it includes the subset of paged pool, pageable operating system code, and pageable driver code that is resident and in the system working set.

  • Size of nonpaged pool View this information by adding the Memory: Pool Nonpaged Bytes counter.

  • Size of the free, zero, and standby lists View the sizes of these lists by adding the Memory: Available Bytes counter. (Use the !memusage kernel debugger command to get the size of each of the lists separately.)

Your graph or report now contains a representation of all of physical memory except for two components that you can't obtain from a performance counter:

  • Nonpaged operating system and driver code

  • The modified and modified-no-write paging lists

Although you can easily obtain the size of both the modified and modified-no-write lists by using the kernel debugger !memusage command, there's no easy way to get the size of the nonpaged operating system and driver code.


     < Day Day Up > 


    Microsoft Windows Internals
    Microsoft Windows Internals (4th Edition): Microsoft Windows Server 2003, Windows XP, and Windows 2000
    ISBN: 0735619174
    EAN: 2147483647
    Year: 2004
    Pages: 158

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net