In the course of this book, we'll be referring to some structures and concepts that might be unfamiliar to some readers. In this section, we'll define the terms we'll be using throughout. You should become familiar with them before proceeding to subsequent chapters.
The Win32 application programming interface (API) is the primary programming interface to the Microsoft Windows operating system family, including Windows 2000, Windows 95, Windows 98, Windows Millennium Edition, and Windows CE. Although we don't describe the Win32 API in this book, we do explain the internal behavior and implementation of key Win32 API functions. For a comprehensive guide to programming the Win32 API, see Jeffrey Richter's book Programming Applications for Microsoft Windows (fourth edition, Microsoft Press, 1999).
Each operating system implements a different subset of Win32. For the most part, Windows 2000 is a superset of all Win32 implementations. The specifics of which services are implemented on which platforms are included in the reference documentation for the Win32 API. This documentation is available for free viewing on line at msdn.microsoft.com and is on the MSDN Library CDROMs. The information in this documentation is also detailed in the file \Program Files\Microsoft Platform SDK\Lib\Win32api.csv (a comma-delimited text file) installed as part of the Platform SDK, which comes with MSDN Professional or can be downloaded for free from msdn.microsoft.com. (See the section "Platform Software Development Kit (SDK)" later in this chapter.)
MSDN stands for Microsoft Developer Network, Microsoft's support program for developers. MSDN offers three CD-ROM subscription programs: MSDN Library, Professional, and Universal. The content of MSDN Library is also available for free on line at the MSDN Web site. For more information, see msdn.microsoft.com.
For the purposes of this book, the Win32 API refers to the base set of functions that cover areas such as processes, threads, memory management, security, I/O, windowing, and graphics. The Win32 API is included as part of the Platform SDK. The internals of the other major categories in the Platform SDK, such as transactions, databases, messaging, multimedia, and networking services, are not covered in this book.
Although Windows 2000 was designed to support multiple programming interfaces, Win32 is the primary, or preferred, interface to the operating system. Win32 has this position because, of the three environment subsystems (Win32, POSIX, and OS/2), it provides the greatest access to the underlying Windows 2000 system services. As we'll explain in Chapter 2, application programs on Windows 2000 don't call native Windows 2000 system services directly—rather, they must use one of the APIs provided by an environment subsystem.
History of the Win32 API
Interestingly, Win32 wasn't slated to be the original programming interface to Microsoft Windows NT. Because the Windows NT project started as a replacement for OS/2 version 2, the primary programming interface was the 32-bit OS/2 Presentation Manager API. A year into the project, however, Microsoft Windows 3.0 hit the market and took off. As a result, Microsoft changed direction and made Windows NT the future replacement for the Windows family of products as opposed to the replacement for OS/2. It was at this juncture that the need to specify the Win32 API arose—before this, the Windows API existed only as a 16-bit interface.
Although the Win32 API would introduce many new functions that hadn't been available on Windows 3.1, Microsoft decided to make the new API compatible with the 16-bit Windows API function names, semantics, and use of data types whenever possible to ease the burden of porting existing 16-bit Windows applications to Windows NT. So those of you who are looking at the Win32 API for the first time and wondering why many function names and interfaces seem inconsistent should keep in mind that one reason for the inconsistency was to ensure that the Win32 API is compatible with the old 16-bit Windows API.
Several terms in the Windows 2000 user and programming documentation have different meanings in different contexts. For example, the word service can refer to a callable routine in the operating system, a device driver, or a server process. The following list describes what certain terms mean in this book:
Although programs and processes appear similar on the surface, they are fundamentally different. A program is a static sequence of instructions, whereas a process is a container for a set of resources used by the threads that execute the instance of the program. At the highest level of abstraction, a Windows 2000 process comprises the following:
A thread is the entity within a process that Windows 2000 schedules for execution. Without it, the process's program can't run. A thread includes the following essential components:
The volatile registers, the stacks, and the private storage area are called the thread's context. Because this information is different for each machine architecture that Windows 2000 runs on, this structure, by necessity, is architecture-specific. In fact, the CONTEXT structure returned by the Win32 GetThreadContext function is the only public data structure in the Win32 API that is machine-dependent.
Although threads have their own execution context, every thread within a process shares the process's virtual address space (in addition to the rest of the resources belonging to the process), meaning that all the threads in a process can write to and read from each other's memory. Threads can't reference the address space of another process, however, unless the other process makes available part of its private address space as a shared memory section (called a file mapping object in the Win32 API) or unless one process opens another process and uses the ReadProcessMemory and WriteProcessMemory functions.
In addition to a private address space and one or more threads, each process has a security identification and a list of open handles to objects such as files, shared memory sections, or one of the synchronization objects such as mutexes, events, or semaphores, as illustrated in Figure 1-1.
Figure 1-1 A process and its resources
Every process has a security context that is stored in an object called an access token. The process access token contains the security identification and credentials for the process. By default, threads don't have their own access token, but they can obtain one, thus allowing individual threads to impersonate the security context of another process—including processes running on a remote Windows 2000 system—without affecting other threads in the process. (See Chapter 8 for more details on process and thread security.)
The virtual address descriptors (VADs) are data structures that the memory manager uses to keep track of the virtual addresses the process is using. These data structures are described in more depth in Chapter 7.
Windows 2000 introduces an extension to the process model called a job. A job object's main function is to allow groups of processes to be managed and manipulated as a unit. A job object allows control of certain attributes and provides limits for the process or processes associated with the job. It also records basic accounting information for all processes associated with the job and for all processes that were associated with the job but have since terminated. In some ways, the job object compensates for the lack of a structured process tree in Windows 2000—yet in many ways is more powerful than a UNIX-style process tree.
You'll find out much more about the internal structure of jobs, processes and threads, the mechanics of process and thread creation, and the thread-scheduling algorithms in Chapter 6.
Windows 2000 implements a virtual memory system based on a flat (linear) 32bit address space. Thirty-two bits of address space translates into 4 GB of virtual memory. On most systems, Windows 2000 allocates half this address space (the lower half of the 4-GB virtual address space, from x00000000 through x7FFFFFFF) to processes for their unique private storage and uses the other half (the upper half, addresses x80000000 through xFFFFFFFF) for its own protected operating system memory utilization. The mappings of the lower half change to reflect the virtual address space of the currently executing process, but the mappings of the upper half always consist of the operating system's virtual memory. Windows 2000 Advanced Server and Datacenter Server support a boot-time option (the /3GB qualifier in Boot.ini) that gives processes running specially marked programs (the large address space aware flag must be set in the header of the executable image) a 3-GB private address space (leaving 1 GB for the operating system). This option allows applications such as database servers to keep larger portions of a database in the process address space, thus reducing the need to map subset views of the database. Figure 1-2 shows the two virtual address space layouts supported by Windows 2000.
Figure 1-2 Address space layouts supported by Windows 2000
Although 3 GB is better than 2 GB, it's still not enough virtual address space to map very large (multigigabyte) databases. To address this need, Windows 2000 has a new mechanism called Address Windowing Extensions (AWE), which allows a 32-bit application to allocate up to 64 GB of physical memory and then map views, or windows, into its 2-GB virtual address space. Although using AWE puts the burden of managing mappings of virtual to physical memory on the programmer, it does solve the immediate need of being able to directly access more physical memory than can be mapped at any one time in a 32-bit process address space. The long-term solution to this address space limitation is 64-bit Windows.
Recall that a process's virtual address space is the set of addresses available for the process's threads to use. Virtual memory provides a logical view of memory that might not correspond to its physical layout. At run time the memory manager, with assistance from hardware, translates, or maps, the virtual addresses into physical addresses, where the data is actually stored. By controlling the protection and mapping, the operating system can ensure that individual processes don't bump into one another or overwrite operating system data. Figure 1-3 illustrates three virtually contiguous pages mapped to three discontiguous pages in physical memory.
Figure 1-3 Mapping virtual memory to physical memory
Because most systems have much less physical memory than the total virtual memory in use by the running processes (2 GB or 3 GB for each process), the memory manager transfers, or pages, some of the memory contents to disk. Paging data to disk frees physical memory so that it can be used for other processes or for the operating system itself. When a thread accesses a virtual address that has been paged to disk, the virtual memory manager loads the information back into memory from disk. Applications don't have to be altered in any way to take advantage of paging because hardware support enables the memory manager to page without the knowledge or assistance of processes or threads.
Details of the implementation of the memory manager, including how address translation works and how Windows 2000 manages physical memory, are described in detail in Chapter 7.
To protect user applications from accessing and/or modifying critical operating system data, Windows 2000 uses two processor access modes (even if the processor on which Windows 2000 is running supports more than two): user mode and kernel mode. User application code runs in user mode, whereas operating system code (such as system services and device drivers) runs in kernel mode. Kernel mode refers to a mode of execution in a processor that grants access to all system memory and all CPU instructions. By providing the operating system software with a higher privilege level than the application software has, the processor provides a necessary foundation for operating system designers to ensure that a misbehaving application can't disrupt the stability of the system as a whole.
The architecture of the Intel x86 processor defines four privilege levels, or rings, to protect system code and data from being overwritten either inadvertently or maliciously by code of lesser privilege. Windows 2000 uses privilege level 0 (or ring 0) for kernel mode and privilege level 3 (or ring 3) for user mode. The reason Windows 2000 uses only two levels is that some of the hardware architectures that were supported in the past (such as Compaq Alpha and Silicon Graphics MIPS) implemented only two privilege levels.
Although each Win32 process has its own private memory space, kernel-mode operating system and device driver code share a single virtual address space. Each page in virtual memory is tagged as to what access mode the processor must be in to read and/or write the page. Pages in system space can be accessed only from kernel mode, whereas all pages in the user address space are accessible from user mode. Read-only pages (such as those that contain executable code) are not writable from any mode.
Windows 2000 doesn't provide any protection to private read/write system memory being used by components running in kernel mode. In other words, once in kernel mode, operating system and device driver code has complete access to system space memory and can bypass Windows 2000 security to access objects. Because the bulk of the Windows 2000 operating system code runs in kernel mode, it is vital that components that run in kernel mode be carefully designed and tested to ensure that they don't violate system security.
This lack of protection also emphasizes the need to take care when loading a third-party device driver, because once in kernel mode the software has complete access to all operating system data. This vulnerability was one of the reasons behind the driver-signing mechanism introduced in Windows 2000, which warns the user if an attempt is made to add an unauthorized (unsigned) driver. (See Chapter 9 for more information on driver signing.) Also, a mechanism called Driver Verifier helps device driver writers to find bugs (such as memory leaks). Driver Verifier is explained in Chapter 7.
As you'll see in Chapter 2, user applications switch from user mode to kernel mode when they make a system service call. For example, a Win32 ReadFile function eventually needs to call the internal Windows 2000 routine that actually handles reading data from a file. That routine, because it accesses internal system data structures, must run in kernel mode. The transition from user mode to kernel mode is accomplished by the use of a special processor instruction that causes the processor to switch to kernel mode. The operating system traps this instruction, notices that a system service is being requested, validates the arguments the thread passed to the system function, and then executes the internal function. Before returning control to the user thread, the processor mode is switched back to user mode. In this way, the operating system protects itself and its data from perusal and modification by user processes.
A transition from user mode to kernel mode (and back) does not affect thread scheduling per se—a mode transition is not a context switch. Further details on system service dispatching are included in Chapter 3.
Thus, it's normal for a user thread to spend part of its time executing in user mode and part in kernel mode. In fact, because the bulk of the graphics and windowing system also runs in kernel mode, graphics-intensive applications spend more of their time in kernel mode than in user mode. An easy way to test this is to run a graphics-intensive application such as Microsoft Paint or Microsoft Pinball and watch the time split between user mode and kernel mode using one of the performance counters listed in Table 1-1.
Table 1-1 Mode-Related Performance Counters
|Processor: % Privileged Time||Percentage of time that an individual CPU (or all CPUs) has run in kernel mode during a specified interval|
|Processor: % User Time||Percentage of time that an individual CPU (or all CPUs) has run in user mode during a specified interval|
|Process: % Privileged Time||Percentage of time that the threads in a process have run in kernel mode during a specified interval|
|Process: % User Time||Percentage of time that the threads in a process have run in user mode during a specified interval|
|Thread: % Privileged Time||Percentage of time that a thread has run in kernel mode during a specified interval|
|Thread: % User Time||Percentage of time that a thread has run in user mode during a specified interval|
Kernel Mode vs. User Mode
You can use the Performance tool to see how much time your system spends executing in kernel mode vs. in user mode. Follow these steps:
- Run the Performance tool by opening the Start menu and selecting Programs/Administrative Tools/Performance.
- Click the Add button (+) on the toolbar.
- With the Processor performance object selected, click the % Privileged Time counter and, while holding down the Ctrl key, click the % User Time counter.
- Click Add, and then click Close.
- Move the mouse rapidly back and forth. You should notice the % Privileged Time line going up when you move the mouse around, reflecting the time spent servicing the mouse interrupts and the time spent in the graphics part of the windowing system (which, as explained in Chapter 2, runs primarily as a device driver in kernel mode). (See Figure 1-4.)
- When you're finished, click the New Counter Set button on the toolbar (or just close the tool).
You can also quickly see this activity by using Task Manager. Just click the Performance tab, and then select Show Kernel Times from the View menu. The CPU usage bar will show user-mode time in green and kernel-mode time in red.
Figure 1-4 Performance tool showing time split between kernel mode and user mode
To see how the Performance tool itself uses kernel time and user time, run it again, but add the individual Process counters % User Time and % Privileged Time for every process in the system:
- If it's not already running, run the Performance tool again. (If it is already running, start with a blank display by pressing the New Counter Set button on the toolbar.)
- Click the Add button (+) on the toolbar.
- Change the Performance Object to Process.
- Select the % Privileged Time and % User Time counters.
- Select all processes in the Instance box (except the _Total process).
- Click Add, and then click Close.
- Move the mouse rapidly back and forth.
- Press Ctrl+H to turn on highlighting mode. This highlights the currently selected counter in white.
- Scroll through the counters at the bottom of the display to identify the processes whose threads were running when you moved the mouse, and note whether they were running in user mode or kernel mode.
You should see the Performance tool process (look in the Instance column for the mmc process) kernel-mode and user-mode time go up when you move the mouse because it is executing application code in user mode and calling Win32 functions that run in kernel mode. You'll also notice kernel-mode thread activity in a process named csrss when you move the mouse. This activity occurs because the Win32 subsystem's kernel-mode raw input thread, which handles keyboard and mouse input, is attached to this process. (See Chapter 2 for more information about system threads.) Finally, the process named Idle that you see spending nearly 100 percent of its time in kernel mode isn't really a process—it's a fake process used to account for idle CPU cycles. As you can observe from the mode in which the threads in the Idle process run, when Windows 2000 has nothing to do, it does it in kernel mode.
In the Windows 2000 operating system, an object is a single, run-time instance of a statically defined object type. An object type comprises a system-defined data type, functions that operate on instances of the data type, and a set of object attributes. If you write Win32 applications, you might encounter process, thread, file, and event objects, to name just a few examples. These objects are based on lower-level objects that Windows 2000 creates and manages. In Windows 2000, a process is an instance of the process object type, a file is an instance of the file object type, and so on.
An object attribute is a field of data in an object that partially defines the object's state. An object of type process, for example, would have attributes that include the process ID, a base scheduling priority, and a pointer to an access token object. Object methods, the means for manipulating objects, usually read or change the object attributes. For example, the open method for a process would accept a process identifier as input and return a pointer to the object as output.
Although there is a parameter named ObjectAttributes that a caller supplies when creating an object using either the Win32 API or native object services, that parameter shouldn't be confused with the more general meaning of the term as used in this book.
The most fundamental difference between an object and an ordinary data structure is that the internal structure of an object is hidden. You must call an object service to get data out of an object or to put data into it. You can't directly read or change data inside an object. This difference separates the underlying implementation of the object from code that merely uses it, a technique that allows object implementations to be changed easily over time.
Objects provide a convenient means for accomplishing the following four important operating system tasks:
Not all data structures in the Windows 2000 operating system are objects. Only data that needs to be shared, protected, named, or made visible to user-mode programs (via system services) is placed in objects. Structures used by only one component of the operating system to implement internal functions are not objects. Objects and handles (references to an instance of an object) are discussed in more detail in Chapter 3.
Windows 2000 supports C2-level security as defined by the U.S. Department of Defense Trusted Computer System Evaluation Criteria (DoD 5200.28-STD, December 1985). This standard includes discretionary (need-to-know) protection for all shareable system objects (such as files, directories, processes, threads, and so forth), security auditing (for accountability of subjects, or users, and the actions they initiate), password authentication at logon, and the prevention of one user from accessing uninitialized resources (such as free memory or disk space) that another user has deallocated.
Windows NT 4 was formally evaluated at the C2 level and is on the U.S. government Evaluated Products List. (Windows 2000 is still in the evaluation process.) Also, Windows NT 4 has met the European organization ITSEC (IT Security Evaluation Criteria) at the FC2/E3 (functional level C2 and assurance level E3, something normally associated only with B-level systems) security level. Achieving a government-approved security rating allows an operating system to compete in that arena. Of course, many of these required capabilities are advantageous features for any multiuser system.
Windows 2000 has two forms of access control over objects. The first form—discretionary access control—is the protection mechanism that most people think of when they think of protection under Windows 2000. It's the method by which owners of objects (such as files or printers) grant or deny access to others. When users log in, they are given a set of security credentials, or a security context. When they attempt to access objects, their security context is compared to the access control list on the object they are trying to access to determine whether they have permission to perform the requested operation.
Privileged access control is necessary for those times when discretionary access control isn't enough. It's a method of ensuring that someone can get to protected objects if the owner isn't available. For example, if an employee leaves a company, the administrator needs a way to gain access to files that might have been accessible only to that employee. In that case, under Windows 2000, the administrator can take ownership of the file so that you can manage its rights as necessary.
Security pervades the interface of the Win32 API. The Win32 subsystem implements object-based security in the same way the operating system does; the Win32 subsystem protects shared Windows objects from unauthorized access by placing Windows 2000 security descriptors on them. The first time an application tries to access a shared object, the Win32 subsystem verifies the application's right to do so. If the security check succeeds, the Win32 subsystem allows the application to proceed.
The Win32 subsystem implements object security on a number of shared objects, some of which were built on top of native Windows 2000 objects. The Win32 objects include desktop objects, window objects, menu objects, files, processes, threads, and several synchronization objects.
For a comprehensive description of Windows 2000 security, see Chapter 8.
If you've worked at all with Windows operating systems, you've probably heard about or looked at the registry. You can't talk much about Windows 2000 internals without referring to the registry because it's the system database that contains the information required to boot and configure the system, systemwide software settings that control the operation of Windows 2000, the security database, and per-user configuration settings (such as which screen saver to use).
In addition, the registry is a window into in-memory volatile data, such as the current hardware state of the system (what device drivers are loaded, the resources they are using, and so on) as well as the Windows 2000 performance counters. The performance counters, which aren't actually "in" the registry, are accessed through the registry functions. See Chapter 5 for more on how performance counter information is accessed from the registry.
Although many Windows 2000 users and administrators will never need to look directly into the registry (since you can view or change most of the configuration settings with standard administrative utilities), it is still a useful source of Windows 2000 internals information because it contains many settings that affect system performance and behavior. (If you decide to directly change registry settings, you must exercise extreme caution; any changes might adversely affect system performance or, worse, cause the system to fail to boot successfully.) You'll find references to individual registry keys throughout this book as they pertain to the component being described. Most registry keys referred to in this book are under HKEY_LOCAL_MACHINE, which we'll abbreviate throughout as HKLM.
For further information on the registry and its internal structure, see Chapter 5.
Windows 2000 differs from most other operating systems in that most internal text strings are stored and processed as 16-bit-wide Unicode characters. Unicode is an international character set standard that defines unique 16-bit values for most of the world's known character sets. (For more information about Unicode, see www.unicode.org as well as the programming documentation in the MSDN Library.)
Because many applications deal with 8-bit (single-byte) ANSI character strings, Win32 functions that accept string parameters have two entry points: a Unicode (wide, 16-bit) and an ANSI (narrow, 8-bit) version. The Windows 95, Windows 98, and Windows Millennium Edition implementations of Win32 don't implement all the Unicode interfaces to all the Win32 functions, so applications designed to run on one of these operating systems as well as Windows 2000 typically use the narrow versions. If you call the narrow version of a Win32 function, input string parameters are converted to Unicode before being processed by the system and output parameters are converted from Unicode to ANSI before being returned to the application. Thus, if you have an older service or piece of code that you need to run on Windows 2000 but this code is written using ANSI character text strings, Windows 2000 will convert the ANSI characters into Unicode for its own use. However, Windows 2000 never converts the data inside files—it's up to the application to decide whether to store data as Unicode or as ANSI.
In previous editions of Windows NT, Asian and Middle East editions were a superset of the core U.S. and European editions and contained additional Win32 functions to handle more complex text input and layout requirements (such as right to left text input). In Windows 2000, all language editions contain the same Win32 functions. Instead of having separate language versions, Windows 2000 has a single worldwide binary so that a single installation can support multiple languages (by adding various language packs). Applications can also take advantage of Win32 functions that allow single worldwide application binaries that can support multiple languages.