Background

The Windows NT series is a family of hybrid microkernel OSs developed and distributed by Microsoft Corporation. It was originally designed through a collaborative effort with IBM as the successor to the OS/2 2.0 Presentation Manager. However, the commercial success of the Windows 3.x series led Microsoft to steer Windows NT development toward its present relationship with the classic Windows API. Therefore, the structure and conventions of the Windows API (Win32) are heavily derived from the original Windows 3.0 API. This influence is so significant that the 1993 release of the original Windows NT was numbered 3.1 to provide parity and a natural transition from the then dominant Windows 3.0. The Windows NT series is currently the flagship product of the Windows line and is simply referred to as "Windows" from here on.

Microsoft Developer Network (MSDN)

The Microsoft Developer Network (MSDN) is the authoritative source of information on Windows APIs and technologies. You'll refer to it regularly over the course of a Windows application security review. A free online version is available at http://msdn.microsoft.com/, and local versions are included with the purchase of Visual Studio or through a subscription-based service.

Windows is termed a hybrid microkernel, but its development history has always shown a willingness to sacrifice the microkernel separation for increased performance. It's probably more accurate to say that it draws from the microkernel design but doesn't fit the definition to an appreciable degree. More appropriately, the basic design of Windows is heavily influenced by the Digital Equipment Corporation (DEC) Virtual Memory System (VMS) operating system because the Windows NT senior architect, David Cutler, had previously worked as one of the primary designers of VMS. Microsoft hired Cutler in 1988 to help develop its next-generation operating system, and he brought a team of former DEC VMS engineers with him.

The combined lineage of VMS and Windows 3.0 gives the modern Windows OS its unique (and occasionally schizophrenic) feel. Accepting some incongruities, the modern Windows system is a highly capable multiuser OS. It's natively multithreaded, all the way down to a fully preemptable kernel. The system provides a flexible security model that allows a fine-grained separation and assignment of resources, which extends to secure authentication across large distributed networks. However, a potential weakness of Windows is that the system supports such a wide range of capabilities. Many historical decisions in designing and implementing these capabilities have created a fertile ground for potential vulnerabilities. Although Microsoft is now one of the most security-aware software companies, the Windows system carries the burden of past security mistakes. It's these idiosyncrasies you need to focus on when considering Windows-specific security vulnerabilities.

This chapter and Chapter 12 provide the information you need to identify vulnerabilities unique to the Windows architecture. Before learning about vulnerabilities, however, you need to understand more about the architecture of the OS. The following sections give you a basic overview of Windows and explain Windows design choices and handling of fundamental OS requirements. This overview isn't comprehensive; it's more a targeted coverage of the details you need to know. However, it should give you the foundation for understanding the types of vulnerabilities covered in this chapter and the next.

Environment Subsystems

The OS market was actually quite volatile when Windows NT was originally designed, so Microsoft chose an interesting approach in designing and implementing its new OS. It implemented the base kernel and user mode interface as one set of components, but the user mode environment and API are actually selectable. They are implemented in environment subsystems; the original Windows NT supported the Portable Operating System Interface for UNIX (POSIX) standard and OS/2 APIs in addition to the core Win32 subsystem. This design allowed Microsoft to hedge its bets and potentially change the top-level operating environment as needed.

The environment subsystem concept never really took off, however, and Win32 effectively cemented itself in the marketplace over time. In response, the bulk of the Win32 subsystem has been migrated into the kernel for improved performance. However, the environment subsystems are still a core underpinning of the OS and provide an interesting architectural point in other contexts.