13. Windows Memory Architecture

Page 209

13. Windows Memory Architecture

		In this chapter, we take a look at how Windows NT and Windows 9x use memory. The primary purpose of this chapter is to give you some background on virtual memory and how it is used. As a rule, Visual Basic programmers don't generally need to manipulate memory directly, but we will see one important reason to do so when we discuss accessing foreign processes. In any case, the subject is both interesting and instructive, so I have decided to cover the highlights in this book.

	Types of Memory

		We should begin by defining our terms. Figure 13-1 shows some of the concepts involved.

		Physical Memory

		Physical memory is the actual RAM that is installed in a PC. Each byte of physical memory has a physical address, which is a number from 0 to one less than the number of bytes of physical memory. For instance, a PC with 64 MB of RAM has physical address &H0000 0000-&H0400 0000 in hex, which is 0 67,108,863 in decimal.

		Physical memory (unlike pagefile and virtual memory discussed later) is executable memory, that is, memory that can be read or written to by the CPU through its instruction set. Thus, for instance, in the assembly language instruction:

		mov [si], ax

		the address in the si register must be a physical address in order for this instruction code to actually execute.

Page 210

		Figure 13-1. Types of memory

		Virtual Memory

		Virtual memory is simply a range of numbers, referred to as virtual addresses. It is nothing more. Thus, while a programmer can refer to a virtual address, Windows cannot access data at that address, since the address does not represent actual physical storage, as do physical and pagefile addresses. In order for code that refers to a virtual address to be executed, the virtual address must be mapped to a physical address, where the code or data can actually exist. This is done by the virtual memory manager (or VMM). We will discuss the operation of the VMM a bit later. The Windows operating system designates certain areas of virtual memory as accessible, that is, accessible to user mode code. All other areas are designated as reserved. Which areas are accessible and which are reserved varies with the operating system version (Windows 9x or Windows NT), as we will see in a moment.

		Memory Page Frames

		As you probably know, the smallest unit of memory that can be addressed is a byte. However, the smallest unit of memory that the Windows VMM deals with is a

Page 211

		memory page, also called a memory page frame. On Intel-style computers, a page frame has size 4KB. We will speak of both virtual pages and physical pages.

		Pagefile Memory

		A pagefile (also called a swap file in Windows 95) is a file that resides on the hard disk. It is used for data and code storage just like physical memory, but is generally larger than physical memory. As we will discuss, Windows uses the pagefile (or pagefiles there can be more than one) to store information that does not fit in RAM swapping pages of information between the pagefile and RAM as needed.

		Thus, a range of virtual addresses can correspond to addresses in a pagefile, rather than to addresses in physical memory (see Figure 13-1). When this happens, we say that the virtual addresses are backed by a pagefile, or pagefile-backed.

		In general, a set of virtual addresses can be backed by physical memory, a pagefile, or indeed any file.

		Memory-Mapped Files

		In Chapter 10, Objects and Their Handles, we briefly discussed memory-mapped files and gave an example of file mapping. Simply put, any file can be used to back virtual memory, just as a pagefile is used in this way. In fact, a pagefile is a file that exists solely for the purpose of backing virtual memory.

		When a file is mapped to memory in this way, the file is called a memory-mapped file. Each of the files in Figure 13-1 is memory-mapped. The corresponding virtual pages are file-backed.

		Recall that the CreateFileMapping function is defined as:

		Private Declare Function CreateFileMapping Lib "kernel32" Alias "CreateFileMappingA" ( _    ByVal hFile As Long, _    ByVal lpSecurityAttributes As Long, _    ByVal flProtect As Long, _    ByVal dwMaximumSizeHigh As Long, _    ByVal dwMaximumSizeLow As Long, _    ByVal lpName As String _ ) As Long

		This function creates a file-mapping object, using the handle of an open file, and returns a handle to the file-mapping object. This handle can be used with the MapViewOfFile function:

		Private Declare Function MapViewOfFile Lib "kernel32" ( _    ByVal hFileMappingObject As Long, _    ByVal dwDesiredAccess As Long, _    ByVal dwFileOffsetHigh As Long, _

Page 212

		ByVal dwFileOffsetLow As Long, _    ByVal dwNumberOfBytesToMap As Long _ ) As Long

		to map the file to virtual memory. The starting address of the file mapping in virtual memory is returned by the MapViewOfFile function. We can also say that the view is backed by the file with handle hFile.

		Note that if the hFile parameter to CreateFileMapping is set to -1, then the file-mapping object (and any views created from this object) are backed by the pagefile, instead of a specified file.

		Shared Physical Memory

		Physical memory is said to be shared when it is mapped into the virtual address space of more than one process, although the virtual addresses may be different in each process. Figure 13-2 illustrates this concept.

		Figure 13-2. Shared physical memory

		If a file, such as a DLL, is in shared physical memory, we may refer to the file as being shared.

Page 213

		One of the advantages of memory-mapped files is that they are easily shared. As we saw in Chapter 10, naming a file-mapping object makes it easy for more than one process to share the file. In this case, the file's contents are mapped to shared physical memory. Figure 13-3 illustrates this. Note that it is also possible to share pagefile contents through the use of file mappings. In particular, we can create a file-mapping object that is backed by a pagefile simply by setting the hFile parameter of the CreateFileMapping function to -1.

		Figure 13-3. A shared file mapping

	The Address Space of a Process

		Every Win32 process gets a virtual address space (also called the address space or process space) of 4GB. Thus, the code within a process can refer to addresses &H0000 0000 through &HFFFF FFFF (or 0 through 232-1 = 4,294,967,295 in decimal). Of course, since a virtual address is just a number, the statement that each process gets its own virtual address space is pretty much meaningless. (It is like saying that each person gets his or her own age range, say from 0 to 150.)

Page 214

		However, this statement is intended to mean that Windows does not draw any relationship between Process A's use of the virtual address &H4000 0000, for example, and Process B's use of the same virtual address &H4000 0000. In particular, Windows may (or may not) associate a different physical address to this virtual address for each process.

		Windows NT and Windows 9x use virtual memory differently.

		Windows NT Address Space Usage

		Figure 13-4 shows the general use of a process address space under Windows NT.

		Region A

		Referring to Figure 13-4, Windows NT reserves the first 64KB of virtual memory for a special purpose and marks it as inaccessible to user mode code. When working with pointers, which VC++ programmers must do quite often, it is easy to forget to initialize a pointer. For instance, consider the following code:

		int *pi;   // declare a pointer to an integer *pi = 5;   // point the pointer to an integer value

		This code will not work. The problem is that a pointer must point to a variable, not a value. The first line of code declares a pointer, which initially contains the value 0. That is, it is a NULL pointer. The second line of code is thus trying to place the value 5 in memory location 0!

		To alert programmers to this commonly made mistake, the lower portion of memory is reserved by Windows (both Windows NT and Windows 9x). As a result, code such as the previous example will cause a GPF, thus alerting the programmer to the error.

		Region B

		Referring to Figure 13-4, region B starts at the 64KB boundary and reaches up to 64 KB short of the 2GB mark. Thus, region B has size 2GB - 128KB (essentially half the entire address space). Portions of this region are mapped to the main EXE for the application, any application-specific DLLs, and system DLLs, such as KERNEL32.DLL. USER32.DLL, GDI32.DLL, and more.

		To illustrate. Figure 13-5 shows the address space of an application called Designer (from Micrografx, Inc.). Note the following:

		The main EXE is located at &H40 0000, which is the default base address for a program written in VC++. (I do not know what development environment was used to write Designer.)

		An application-specific DLL called DS70RES.DLL is located at &H0135 0000, which is somewhat above the EXE.

Page 215

		Figure 13-4. Process address space usage under Windows NT

		The next module (rpcltc.dll) at &H015B 0000 is a Microsoft DLL.

		There follow several Micrografx and Microsoft DLLs and even a Creative Labs DLL (sb16snd.dll)!

		The rest of the DLLs, starting with COMCTR32.DLL, are copyrighted by Microsoft. This includes USER32.DLL at &H77E7 0000, GDI32.DLL at &H77ED 0000, and KERNEL32.DLL at &H77F0 0000.

Page 216

		Figure 13-5. An example process space

		It is worth remarking that this region may contain both shared and unshared files. For instance. Designer's application-specific DLLs are likely to be unshared (unless other copies of Designer are running). On the other hand, the system DLLs, such as KERNEL32.DLL. are shared, in that there is only one copy of these DLLs in physical memory. Generally, this copy is mapped to the same virtual addresses in each process, but this is not a requirement.

		Region C

		Referring to Figure 13-4, region C is the 64KB region just below the 2GB

		mark, where Windows reserved address space begins. It is used as a kind of

		barrier that prevents a programmer from using memory addresses that straddle

		the accessible region B and the inaccessible region D which belongs to

		Windows.

Page 217

		Region D

		This region is reserved for use by Windows NT (the executive, kernel, and device drivers, for example). Applications cannot access memory in this virtual address range. Any attempt to do so will result in an access violation (a GPF).

		Windows 9x Address Space Usage

		Figure 13-6 shows the general use of a process address space under Windows 9x.

		Region A

		Referring to Figure 13-6, Windows 9x reserves region A, which is a mere 4 KB in size, for the same purpose as Windows NT reserves the first 64 KB of memory as an alert for null pointers. This region is protected and will cause an access violation if we attempt to access this memory in user mode code.

		Region B

		The next region of memory (region B) is used to maintain compatibility with DOS and 16-bit Windows applications. Although it is accessible, a programmer should not use this region.

		Region C

		Region C is the address space used by application code and its DLLs. Also, Windows modules are placed in this region. For instance, if an application requires a custom control (OCX), the module will be located in this region.

		Region D

		Windows 9x maps the Win32 system DLLs (KERNEL32.DLL, USER32.DLL, and so on) into this address space. These files are shared, meaning that more than one process has access to a single copy of the files in physical memory.

		This region of memory is accessible to user mode code (although I certainly would not recommend doing so).

		Region E

		This region also contains shared Windows files, such as the Windows executive and kernel, virtual device drivers, file system, and memory management code, and so on.

		This region of memory is also accessible to user mode code.

Page 218

		Figure 13-6. Process address space usage under Windows 9x

	Example: Using GetSystemInfo

		The GetSystemInfo API function retrieves some information about address spaces. Its syntax is:

		VOID GetSystemInfo(    LPSYSTEM_INFO lpSystemInfo // address of system information structure );

Page 219

		or, in VB:

		Declare Sub GetSystemInfo Lib "kernel32" Alias "GetSystemInfo" ( _    lpSystemInfo As SYSTEM_INFO)

		The system information structure is defined as:

		struct _SYSTEM_INFO {    union { DWORD dwOemId;       struct {          WORD wProcessorArchitecture;          WORD wReserved;       };    };    DWORD dwPageSize;    LPVOID lpMinimumApplicationAddress;    LPVOID lpMaximumApplicationAddress;    DWORD dwActiveProcessorMask;    DWORD dwNumberOfProcessors;    DWORD dwProcessorType;    DWORD dwAllocationGranularity;    WORD wProcessorLevel;    WORD wProcessorRevision; }

		A union is a VC++ construct that means that the variable in question can hold more than one data type at different times. For instance, a variable whose type is the following union:

		union uExample {    int i;    char ch; }

		can be either an integer or a character. Of course, the system must set aside enough space for the largest possibility, in this case, 4 bytes for a VC++ integer. Naturally, we must do the same.

		The SYSTEM_INFO structure contains a union whose members are a DWORD and another structure consisting of two WORDs. Thus, the size of the union is that of a DWORD (or two WORDs), which is 4 bytes.

		We can define this structure in VB as:

		Type SYSTEM_INFO    wProcessorArchitecture As Long    dwPageSize as Long    lpMinimumApplicationAddress As Long    lpMaximumApplicationAddress As Long    dwActiveProcessorMask As Long    dwNumberOfProcessors As Long    dwProcessorType As Long    dwAllocationGranularity As Long    wProcessorLevel As Integer    wProcessorRevision As Integer End Type

Page 220

		Here is some code that calls GetSystemInfo, along with its output on my system:

		Public Sub GetSystemInfoExample() Dim si As SYSTEM_INFO GetSystemInfo si Debug.Print "Page size: " & si.dwPageSize Debug.Print "Max app address: " & Hex(si.lpMaximumApplicationAddress) Debug.Print "Min app address: " & Hex(si.lpMinimumApplicationAddress) Debug.Print "Application Granularity: " & si.dwAllocationGranularity Debug.Print "Processor architecture: " & si.wProcessorArchitecture Debug.Print "Processor count: " & si.dwNumberOfProcessors Debug.Print "Processor type: " & si.dwProcessorType Debug.Print "Processor level: " & si.wProcessorLevel Debug.Print "Processor revision: " & Hex(si.wProcessorRevision) End Sub Page size: 4096 Max app address: 7FFEFFFF Min app address: 10000 Application Granularity: 65536 Processor architecture: 0 Processor count: 1 Processor type: 586 Processor level: 6 Processor revision: 501

		This reports, as expected, that the page size is 4KB. Also, under Windows NT (which I am running), the minimum address for an application is &H1000 (= 64 KB) as expected from Figure 13-4. Similarly, the maximum application address is &H7FFE FFFF, as expected.

		We will explain application granularity later.

		According to the documentation, the wProcessorArchitecture variable is set to 0 to represent PROCESSOR_ARCHITECTURE_INTEL. (The dwOemID parameter is obsolete.) The processor level of 6 indicates a Pentium II processor. The processor revision number 501 is interpreted as Model 5 stepping 1. (For more details, please see the GetSystemInfo documentation.)

	Allocating Virtual Memory

		Each page of virtual address space can be in one of three states:

		Reserved The page has been reserved for use.

		Committed Physical storage is allocated for this virtual page. The storage is either in a pagefile or in a memory-mapped file.

Page 221

		Free The page is neither reserved nor committed and is therefore currently not accessible to the process.

		Virtual memory can be reserved or committed by calling the API function VirtualAlloc:

		LPVOID VirtualAlloc(    LPVOID lpAddress,       // address of region to reserve or commit    DWORD dwSize,           // size of region    DWORD flAllocationType, // type of allocation    DWORD flProtect         // type of access protection );

		The flAllocationType parameter can be set to the following constants (among others):

		MEM_RESERVE This simply reserves a range of the process's virtual address space without allocating any physical storage. (Storage can be allocated with a subsequent call to this function, however.)

		MEM_COMMIT This allocates physical storage in memory or in the paging file on disk for the specified reserved region of pages.

		The two constants can be combined in order to reserve and commit in one operation.

		Separating the process of reserving and committing memory has some advantages. For example, since reserving memory is a very efficient process, if an application should require a large amount of memory, it can reserve the entire amount but commit only the portion that it currently requires, thus spreading out the more time-consuming process of commitment of physical storage.

		In fact, Windows itself uses this approach when it allocates a stack for each newly created thread. Windows reserves 1MB of virtual memory for each thread's stack but initially commits only two pages (8KB).

		Memory protection

		Note that the flProtect parameter of VirtualAlloc is used to specify the type of access protection to accord the newly committed virtual memory (this does not apply to reserved memory). The choices are:

		PAGE_READONLY Gives read-only access to the committed virtual memory.

		PAGE_READWRITE Gives read-write access to the committed virtual memory.

Page 222

		PAGE_WRITECOPY Gives copy-on-write access to the committed virtual memory. We will discuss this a bit later.

		PAGE_EXECUTE Enables execute access to the committed virtual memory. Any attempt to read or write to the memory, however, will result in an access violation.

		PAGE_EXECUTE_READ Enables execute and read access.

		PAGE_EXECUTE_READWRITE Enables execute, read, and write access.

		PAGE_EXECUTE_WRITECOPY Enables execute, read, and copy-on-write access.

		PAGE_NOACCESS Disables all access to the committed virtual memory.

		In addition. any of these values except PAGE_NOACCESS can be ORed with the following two flags.

		PAGE_GUARD Designates the pages as guard pages. Any attempt to access a guard page causes the system to raise a STATUS_GUARD_PAGE exception and turn off the guard page status. Thus, guard pages raise a first-access-only alarm.

		PAGE_NOCACHE Prevents caching of the committed memory.

		Let us explain copy-on-write. Suppose that a certain page of physical memory is being shared by two processes. If the page is marked as read-only, then the two processes can share the page without problems. However, there may be situations in which we want to permit each process to write to this memory but not to affect the other process. By setting the protection to copy-on-write, as soon as one process attempts to write to the shared page, the system makes a copy of that page exclusively for the process that wants to do the writing. Thus, the page is no longer shared and the other process's view of the data remains unchanged.

		We note also that the protection attributes of a page can be changed using the VirtualProtect API function.

		Allocation granularity

		Note that if the lpAddress parameter is not a multiple of 64KB, then the system will round down the requested address to the nearest multiple of 64KB. Windows always reserves virtual memory starting on an allocation granularity boundary, which, in the case of Intel-style processors, is a multiple of 64KB. In other words, the starting address of any block of reserved memory is a multiple of 64KB.

Page 223

		In addition, the amount of memory that is allocated is always a multiple of the system's page size, that is, of 4KB. Hence, VirtualAlloc will round up any memory request to the nearest multiple of the page size.

		Incidentally, there are times when Windows allocates virtual memory on behalf of a process, but for Windows' own use. In these cases, Windows does not always follow the rules about allocation granularity boundaries that it follows when allocating memory requested by a programmer (although page size requirements are honored).

		The Virtual Address Descriptor

		Finally, in case you are wondering, the system keeps track of which virtual pages are reserved in a structure called a Virtual Address Descriptor or VAD. This is essential because there is no other way to tell when virtual addresses have been reserved.

	Example: Using GlobalMemoryStatus

		The API function GlobalMemoryStatus:

		Declare Sub GlobalMemoryStatus Lib "kerne132" Alias "GlobalMemoryStatus" ( _    lpBuffer As MEMORYSTATUS)

		reports on a variety of memory-related values by filling in the following structure:

   struct _MEMORYSTATUS {        DWORD dwLength;         // sizeof(MEMORYSTATUS)        DWORD dwMemoryLoad;     // percent of memory in use        DWORD dwTotalPhys;      // bytes of physical memory        DWORD dwAvailPhys;      // free physical memory bytes        DWORD dwTotalPageFile;  // bytes of paging file        DWORD dwAvailPageFile;  // free bytes of paging file        DWORD dwTotalVirtual;   // user bytes of address space        DWORD dwAvailVirtual;   // free user bytes      }       

		In VB, this is:

   Type MEMORYSTATUS        dwLength  As Long            sizeof(MEMORYSTATUS)        dwMemoryLoad  As Long        percent of memory in use        dwTotalPhys  As Long         bytes of physical memory        dwAvailPhys  As Long         free physical memory bytes        dwTotalPageFile  As Long     bytes of paging file        dwAvailPageFile  As Long     free bytes of paging file        dwTotalVirtual  As Long      user bytes of address space        dwAvailVirtual  As Long      free user bytes      End Type       

Page 224

		For example, the code:

		Dim ms As MEMORYSTATUS GlobalMemoryStatus ms Debug.Print "Virtual memory: " & ms.dwTotalVirtual Debug.Print "Short of 2 GB: " & (2 ^ 31 - ms.dwTotalVirtual) / 1024 & " KB" Debug.Print "Available Virtual: " & ms.dwAvailVirtual

		might produce:

		Virtual memory: 2147352576 Short of 2 GB: 128 KB Available Virtual: 1965658112

		Note that the available virtual memory is 128KB less than the 2GBs in the lower half of the total address space of a process, as we would expect from Figure 13-4.

	Virtual Memory Management

		We want to take a look next at how the Windows virtual memory manager (abbreviated VMM) actually interprets virtual memory addresses as physical addresses.

		Translating Virtual Addresses to Physical Addresses: Page Hits

		Figure 13-7 shows the translation process when the virtual address is mapped to a physical address. This situation is referred to as a (physical) page hit.

		Figure 13-7. Virtual-to-physical address resolution for a page hit

		All virtual addresses are divided into three parts. The leftmost part (bits 22 31) contains an index into the process's page directory. Windows maintains a separate

Page 225

		page directory for each process. The address of this page directory is placed in a CPU register called CR3. (Incidentally, part of the task-switching process is to switch this register so that it points to the incoming process's page directory.) A page directory contains 1024 4-byte entries.

		Windows also maintains, for each process, a collection of page tables. Each page directory entry contains a page table number, which identifies a particular page table. Thus, Windows supports up to 1024 page tables. (Actually, page tables are created only when needed, that is, when an attempt is made to access data or code at a given virtual address, not simply when the virtual memory is allocated.)

		The second part of the virtual address (bits 12 21) is used as an index into the page table corresponding to the page directory entry. The page table entry at this index contains, in its leftmost 20 bits, a page frame number that identifies a particular page frame in physical memory.

		The third and final part of the virtual address (bits 0 11) is an offset into that page frame. The combination of page frame number and offset into that page frame constitutes a physical address in physical memory.

		Since each page table has 1024 entries and there are 1024 page tables, the total number of page frames that can be identified in this manner is 1024 1024 = 210 210 = 220. Since each page frame is 4KB = 4 210 bytes in size, the theoretical limit of physical addresses is 4 230 = 4GB.

		This rather complex-looking translation scheme has several important advantages. One immediate advantage is that page frames are very small and can easily be kept in memory. It is much easier to find 4KB of consecutive physical memory than, say, 64KB.

		However, the main advantage is that a virtual memory address from two processes can be deliberately made to point to different physical addresses or be deliberately made to point to the same physical address!

		To illustrate, suppose that Process1 and Process2 both refer in code to the same virtual address. Process1's virtual-to-physical address resolution uses its page directory, and Process2's translation uses its page directory. Thus, while the indices into the page directories are the same in both cases, these indices are indices into different directories. In this way, the VMM can assure that the virtual addresses in each process resolve to different physical addresses.

		On the other hand, the VMM can also ensure that virtual addresses in the two processes, whether they be the same virtual address or not, can resolve to the same physical address. One way to do this would be to have both page directory entries point to the same page table, and thus to the same page frames. In this way, processes can share physical memory.

Page 226

		System page directory and page tables

		We should also mention that Windows maintains a system page directory to deal with virtual memory reserved for Windows, as well as a corresponding set of system page tables.

		Format of a valid page table entry

		A page table entry is said to be valid when it points to a physical page frame, as in Figure 13-7. Figure 13-8 shows the format of a valid page table entry (some data is omitted).

		Figure 13-8. Valid page table entry

		Note that a page table entry contains flags that describe various properties of the page frame, such as whether or not it has been written to or read from and whether or not access is permitted to this memory from user mode code. The least significant bit is the valid bit (also called a present bit), signaling that this entry is valid.

		Translating Virtual Addresses to Physical Addresses: Page Faults

		If a virtual page is not backed by physical memory, but is backed instead by a pagefile, then the translation process will come a cropper, resulting in a page fault. The situation is pictured in Figure 13-9.

Page 227

		Figure 13-9. Virtual-to-physical address resolution for a page fault

		In this case, the system will see that the page table entry is invalid, having its least significant bit set to 0. The format of an invalid page table entry whose page is backed by a pagefile is shown in Figure 13-10.

		Figure 13-10. A page table entry backed by a pagefile

		The fact that the valid bit is 0 tells the system that the page table entry contains a pagefile number between 0 and 15 that identifies the pagefile that backs the virtual address. The page table entry also contains the number of the page within the pagefile that backs the virtual address. The system can then copy that pagefile page into physical memory, change the page table entry from an invalid one to a valid one, and use the 12 bits of the virtual address as an offset into the physical page frame, just as before. This process is referred to as paging.

		Shared Pages

		Our discussion so far concerns unshared physical memory only. The situation for shared physical memory is considerably more complex, and we will not go into the details here, except to say that the VMM uses a concept called prototype page table entries. The idea is simply to point the ordinary page table entry in each process

Page 228

		with shared memory to a single prototype page table entry, instead of to physical memory. In this way, a single prototype page table entry can point to shared physical memory.

		Working Sets

		We have discussed the fact that each virtual page in the 4GB virtual address space of a process exists in one of three states free, reserved, or committed. We can now also say that each committed page is either valid or invalid. Those pages that are valid, that is, backed by physical memory, are referred to collectively as the working set of the process. Of course, the working set is constantly changing, as pages are paged in and out of physical memory. Recall that the rpiEnumProcsNT utility also displays the current size of the working set for a process.

		The system working set describes the virtual pages of system memory that are currently mapped to physical memory.

		The size of a process's working set is limited by minimum and maximum settings that Windows makes, based on the size of physical memory. The settings are described in Table 13-1.

Table 13-1. Process Working Set Size Limits
Memory Model	Memory Size	Minimum Process Working Set Size	Maximum Process Working Set Size
Small	<=19MB	20 pages (80KB)	45 pages (180KB)
Medium	20 32MB	30 pages (120KB)	145 pages (580KB)
Large	>=32MB	50 pages (200KB)	345 pages (1380KB)

		Note that these limits can be changed using the API function SetProcessWorkingSetSize:

   BOOL SetProcessWorkingSetSize(        HANDLE hProcess,                    // open handle to the process of interest     DWORD dwMinimumWorkingSetSize,      // specifies minimum working set size in bytes        DWORD dwMaximumWorkingSetSize,      // specifies maximum working set size in bytes      );       

		Incidentally, setting both of the size parameters to -1 will cause the function to reduce the working set to size 0, thus temporarily removing the process from physical memory.

		We also note that the actual size of a process's working set will vary with time, since Windows will increase the working set if it notices that the process is incurring a large number of page faults.

		The system working set size limits are described in Table 13-2.

Page 229

Table 13-2. System Working Set Size Limits
Memory Model	Memory Size	Minimum Process Working Set Size	Maximum Process Working Set Size
Small	<=19MB	388 pages (1.5MB)	500 pages (2.0MB)
Medium	20 32MB	688 pages (2.7MB)	1150 pages (4.5MB)
Large	>=32MB	1188 pages (4.6MB)	2050 pages (8MB)

		The Page Frame Database

		Windows keeps track of the state of each physical page of memory in a data structure called the Page Frame Database. Each physical page can be in one of eight different states:

		Active (or valid) This page is currently mapped to virtual memory. Thus, it is the target of a working set page.

		Transition On its way to becoming an active page.

		Standby The page was just removed from active status, but it was not modified while active.

		Modified The page has been removed from active status. Its contents were modified while it was active, but have not yet been written to disk.

		Modified No Write Similar to modified but marked so that the contents will not be flushed to disk. Used by the Windows file system drivers.

		Free The page is free but has been written to and is thus not eligible for use by a process.

		Zeroed The page is free and initialized to all 0s by the zero page thread. This page can be allocated to a process.

		Bad The page has incurred parity errors or some other hardware problem and will not be used.

	Heaps of Memory

		Simply put, a heap is a region of virtual memory that has been reserved by Windows. The Win32 API supplies functions for allocating memory from a heap,

Page 230

		freeing heap memory, and resizing blocks of allocated memory. In short, these functions provide a more intuitive method for manipulating application memory than VirtualAlloc and are especially suited for manipulating a large number of small blocks of memory. Although VB programmers don't have any direct use for heaps, the concept does appear quite often in documentation, so we should discuss it briefly.

		Heaps Under 32-Bit Windows

		When a process is created, Windows assigns it a default heap. That is, Windows initially reserves a 1MB region of virtual memory. However, the system will adjust the size of the default heap as necessary. The default heap is used by Windows itself in a variety of ways. For instance, as discussed in Chapter 6, Strings, Visual Basic translates Unicode strings to ANSI for passing to the ANSI entry points of a DLL. Memory for these temporary ANSI strings is obtained from the default heap. The API function GetProcessHeap is used to obtain a handle to the default heap. A programmer can also create additional heaps using the HeapCreate function, which returns a heap handle.

		There are several reasons why a programmer might want to create additional heaps rather than using the default heap. For example, heaps that are dedicated to a particular task are often more efficient. Also, errors made in writing data to heap memory allocated from a dedicated heap will not affect data in other heaps. Finally, allocating memory from a dedicated heap will generally mean that the data is packed more closely together in memory, which may reduce the need for paging data from the pagefile. We should also mention that access to a heap is serialized, that is, the system makes each thread attempting access to heap memory wait its turn until other threads are finished. In this way, only one thread at a time can allocate or free heap memory, thus avoiding unpleasant conflicts.

		Incidentally, the 16-bit version of Windows supported both a global heap and a local heap. Accordingly, 16-bit Windows implemented the functions GlobalAlloc and LocalAlloc. These functions still work, but are not very efficient and should be avoided under Win32. (We will see, however, that they still must be used for certain purposes, such as creating a Clipboard viewer.)

		Heap Functions

		The following functions are used with heaps:

		GetProcessHeap Returns a handle to the default heap for the process.

		GetProcessHeaps Returns a list of the handles for all heaps

		currently used by a process.

Page 231

		HeapAlloc Allocates a block of memory from the specified heap.

		HeapCompact Defragments a heap by combining free blocks. It can also decommit unused pages of heap memory.

		HeapCreate Creates a new heap in the process's address space.

		HeapDestroy Destroys the specified heap.

		HeapFree Frees a previously allocated heap memory block.

		HeapLock Locks a heap so that only a single thread has access to the heap. Other threads requesting access are suspended until the owning thread unlocks the heap. This provides a form of thread synchronization. This is also how the system enforces thread serialization.

		HeapReAlloc Reallocates a block of memory on the heap. Used to change a block's size.

		HeapSize Returns the size of an allocated heap memory block.

		HeapUnlock Unlocks a heap that was previously locked using the HeapLock function.

		HeapValidate Verifies that a heap (or block within a heap) is valid; that is, that it has not been corrupted.

		HeapWalk Allows the programmer to walk the contents of a heap. Used mostly for debugging purposes.

	Example: Mapping Virtual Memory

		The Win32 API function VirtualQuery can be used to get information about the state of virtual memory addresses. The syntax is:

  DWORD VirtualQuery(       LPCVOID lpAddress,                   // address of region       PMEMORY_BASIC_INFORMATION lpBuffer,  // address of information buffer       DWORD dwLength                       // size of buffer     );       

		In VB, this can be declared as:

		Declare Function VirtualQuery Lib "kernel32" (    ByVal lpAddress As Long, _

Page 232

		lpBuffer As MEMORY_BASIC_INFORMATION, _    ByVal dwLength As Long _ ) As Long

		There is also VirtualQueryEx, an extended version of VirtualQuery that permits looking at foreign virtual address spaces:

   DWORD VirtualQueryEx(        HANDLE hProcess,                    // handle to process        LPCVOID lpAddress,                  // address of region        PMEMORY_BASIC_INFORMATION lpBuffer, // address of information buffer        DWORD dwLength                      // size of buffer      );       

		or, in VB:

		Declare Function VirtualQueryEx Lib "kernel32" ( _    ByVal hProcess As Long, _    ByVal lpAddress As Long, _    lpBuffer As MEMORY_BASIC_INFORMATION, _    ByVal dwLength As Long _ ) As Long

		As expected, the hProcess parameter requires a process handle. The lpAddress parameter is the starting address on which to report, but it will be rounded down to the closest multiple of the page size (4KB). Both functions return information in the structure:

  struct _MEMORY_BASIC_INFORMATION {       PVOID BaseAddress;       // base address of region       PVOID AllocationBase;    // allocation base address       DWORD AllocationBase;    // initial access protection       DWORD RegionSize;        // size, in bytes, of region       DWORD State;             // committed, reserved, free       DWORD Protect;           // current access protection       DWORD Type;              // type of pages     }       
 

		or, in VB:

   Type MEMORY_BASIC_INFORMATION        BaseAddress As Long          base address of region        AllocationBase As Long       allocation base address        AllocationProtect As Long    initial access protection        RegionSize As Long           size, in bytes, of region        State As Long                committed, reserved, free        Protect As Long              current access protection        Type As Long                 type of pages      End type    

		To understand the members of this structure, we need to describe what the function does. Unfortunately, the documentation does a rotten job of explaining this function, but it appears to perform as follows. To aid our discussion, let us refer to the page containing the address lpAddress as the specified page. Figure 13-11 will help define our terms.

Page 233

		Figure 13-11. The VirtualQuery(Ex) function

		The following members of MEMORY_BASIC_INFORMATION are always set by VirtualQuery(Ex):

		BaseAddress Returns the base address of the specified page.

		RegionSize The number of bytes from the start of the specified page to the top of the specified region. Despite the name, this is not generally the size of the entire specified region. We will define the term specified region in a moment.

		If the page containing lpAddress is free (neither reserved nor committed), the State member contains the symbolic constant MEM_FREE. No other member (except BaseAddress and RegionSize) has meaning.

		If the page containing lpAddress is not free, the function identifies the allocation region containing the specified page, that is, the region of virtual memory that contains the specified page and was originally allocated by a call to VirtualAlloc.

		Starting at the base address of the specified page, the function examines each successive page in the allocation region, checking to see if it matches the specified page in allocation type and protection type. The collection of all consecutive matching pages is the specified region. The values in the MEMORY_BASIC_INFORMATION structure refer to this region. Specifically, a page matches the specified page if it satisfies the following two conditions.

Page 234

		The page has the same allocation type as the initial page, as reflected by the flag values: MEM_COMMIT, MEM_RESERVE, MEM_FREE, MEM_PRIVATE, MEM_MAPPED, or MEM_IMAGE.

		The page has the same protection type as the initial page as reflected by the flag values: PAGE_READONLY, PAGE_READWRITE, PAGE_NOACCESS, PAGE_WRITECOPY. PAGE_EXECUTE, PAGE_EXECUTE_READ, PAGE_EXECUTE_READWRITE. PAGE_EXECUTE_WRITECOPY, PAGE_GUARD, or PAGE_NOCACHE.

		Now we can describe the other members of the MEMORY_BASIC_INFORMATION structure:

		AllocationBase The base address of the allocation region.

		AllocationProtect The initial protection type of the allocation region.

		State Either MEM_FREE, MEM_RESERVE, or MEM_COMMIT. Refers to the specified region.

		Protect The current protection type of the specified region.

		Type Either MEM_IMAGE, MEM_MAPPED, or MEM_PRIVATE. Refers to the specified region. The meanings of these constants are as follows: MEM_IMAGE indicates that the region is mapped to an image (that is, executable) file; MEM_MAPPED indicates that the region is mapped to a nonexecutable memory-mapped file (such as a data file); MEM_PRIVATE indicates that the region is private, that is, not shared by more than one process.

		Finally, we note that these functions return the size of the MEMORY_BASIC_INFORMATION structure.

		The following procedure fills a list box with a virtual memory map. You will find this procedure in the rpiEnumProcsNT and rpiEnumProcs95 applications on the CD.

		Sub MapMemory(lProcessID As Long) Dim hProcess As Long Dim lret As Long Dim meminfo As MEMORY_BASIC_INFORMATION Dim lCurRegion As Long Dim sName As String Dim sItem As String Dim Tabstops (1 To 5) As Long Dim bIsFree As Boolean

Page 235

		lstMain.FontName = "Courier New" lstMain.FontSize = 9 ' Get process handle from process ID hProcess = ProcHndFromProcID(lProcessID) If hProcess = 0 Then Stop lstMain.Clear ' Set tab stops for list box Tabstops(1) = 4 * 22 Tabstops(2) = Tabstops(1) + 4 * 10 Tabstops(3) = Tabstops(2) + 4 * 10 Tabstops(4) = Tabstops(3) + 4 * 12 SendMessage lstMain.hWnd, LB_SETTABSTOPS, 5, Tabstops(1) Do    ' Get info on this region    lret = VirtualQueryEx(hProcess, lCurRegion, meminfo, LenB(meminfo)) ' Is it free?    bIsFree = (meminfo.State = MEM_FREE)    ' Build item for list box    sItem = Hex(lCurRegion) & " - " & Hex(lCurRegion + meminfo.RegionSize - 1) _        & vbTab & meminfo.RegionSize / 4096 & " pp" _        & vbTab & MemType(meminfo.State)    If Not bIsFree Then         sItem = sItem & vbTab & MemType(meminfo.Type) _    m302 1       & vbTab & AccessType(meminfo.Protect)    End If    ' Get module name. Skip 0, since this gives EXE name.    If lCurRegion > 0 And Not bIsFree Then       sName = String$(MAX_PATH + 1, 0)       lret = GetModuleFileName(lCurRegion, sName, MAX_PATH)       If lret <> 0 Then sItem = sItem & vbTab & Trim0 (sName)    End If         ' Display    lstMain.AddItem sItem       ' Next start of region    lCurRegion = lCurRegion + meminfo.RegionSize       If lCurRegion >= &H7FFEFFFF Then Exit Do Loop    CloseHandle hProcess    End Sub

Page 236

		A portion of the output for one process is shown here:

		0 - FFFF 16 pp FREE

		10000 - 10FFF

		1 pp

		COMMIT

		PRIVATE

		READWRIRE

		11000 - 1 FFFF

		15 pp

		FREE

		20000 - 20FFF

		1 pp

		COMMITE

		PRIVATE

		READWRITE

		21000 - 2FFFF

		15 pp

		FREE

		30000 - 12DFFF

		253 pp

		RESERVE

		PRIVATE

		0

		12D000 - 12FFFF

		1 pp

		COMMIT

		PRIVATE

		260

		12E000 - 12FFFF

		2 pp

		COMMIT

		PRIVATE

		READWRITE

		130000 - 130FFF

		1 pp

		COMMIT

		PRIVATE

		READWRITE

		131000 - 13FFFF

		15 pp

		FREE

		140000 - 158FFF

		25 pp

		COMMITE

		PRIVATE

		READWRITE

		159000 - 23FFFF

		231 pp

		RESERVE

		PRIVATE

		0

		240000 - 240FFF

		1 pp

		COMMIT

		MAPPED

		READWRITE

		241000 - 24FFFF

		15 pp

		RESERVE

		MAPPED

		0

		250000 - 265FFFF

		22 pp

		COMMIT

		MAPPED

		READONLY

		266000 - 26FFFF

		10 pp

		FREE

		270000 - 293FFF

		36 pp

		COMMIT

		MAPPED

		READONLY

		294000 - 29FFFF

		12 PP

		FREE

		2A0000 - 2E0FFF

		65 pp

		COMMIT

		MAPPED

		READONLY

		2E1000 - 2EFFFF

		15 pp

		FREE

		2F0000 - 2F2FFF

		3 pp

		COMMIT

		MAPPED

		READONLY

		2F3000 - 2FFFFF

		13 pp

		FREE

		300000 - 305FFF

		6 pp

		COMMIT

		MAPPED

		EXECUTE_READ

		306000 - 3BFFFF

		186 pp

		RESERVE

		MAPPED

		0

		3C0000 - 3C0FFF

		1 pp

		COMMIT

		MAPPED

		EXECUTE_READ

		3C1000 - 3C7FFF

		7 pp

		RESERVE

		MAPPED

		0

		3C8000 - 3FFFFF

		56 pp

		400000 - 400FFF

		1 pp

		COMMIT

		IMAGE

		READONLY G:\Visual Studio\VB98\VB6.

		EXE

		401000 - 40BFFF

		11 pp

		IMAGE

		EXECUTE_READ

		40C000 - 40DFFF

		2 pp

		COMMIT

		IMAGE

		READWRITE

		Two utility functions used in MapMemory are:

		Function MemType(vValue As Variant) As String ' Function returns name of constant for given value. Dim sName As String Select Case vValue    Case &H1000       sName = "COMMIT"    Case &H2000       sName = "RESERVE"    Case &H4000       sName = "DECOMMIT"    Case &H8000       sName = "RELEASE"    Case &H10000       sName = "FREE"    Case &H20000       sName = "PRIVATE"    Case &H40000       sName = "MAPPED"    Case &H80000       sName = "RESET"

Page 237

		Case &H100000       sName = "TOP_DOWN"    Case &H1000000       sName = "IMAGE"    Case Else       sName = vValue End Select MemType = sName End Function ' ------------------ Function AccessType(vValue As Variant) As String ' Function returns name of constant for given value. Dim sName As String Select Case vValue    Case &H1       sName = "NOACCESS"    Case &H2       sName = "READONLY"    Case &H4       sName = "READWRITE"    Case &H8       sName = "WRITECOPY"    Case &H10       sName = "EXECUTE"    Case &H20       sName = "EXECUTE_READ"    Case &H40       sName = "EXECUTE_READWRITE"    Case &H80       sName = "EXECUTE_WRITECOPY"    Case &H100       sName = "GUARD"    Case &H200       sName = "NOCACHE"    Case Else       sName = vValue End Select AccessType = sName End Function