Getting Started with Driver Testing

If you are new to driver testing, you'll need to set up a test system in addition to your development system. This section introduces basic system requirements and provides an overview of the testing tools that Microsoft provides.

If testing your driver uncovers bugs, see Chapter 22, "How to Debug WDF Drivers," for details about debugging KMDF and UMDF drivers.

Choosing a Test System

To test KMDF drivers, you need a test system that is separate from your development system. This allows the debugger to run without interfering with normal operation of Windows, which would affect the behavior of the driver you are testing. It also saves time rebooting and recovering from the system crashes that are an inevitable part of the development process.

You can test and debug a UMDF driver on a single system, without having a separate system for the debugger.

 Recommended test system capabilities  A test system should have the following capabilities:

• A serial port or IEEE 1394 port for kernel-mode debugging.

• A multiprocessor or at least a hyperthreaded system, unless for some reason you are targeting your driver for only uniprocessor systems.

  You cannot debug multiprocessor issues on a uniprocessor system. Now that even low-end systems support hyperthreading, it is essential to test your driver for multiprocessor scenarios. A multiprocessor x64 system makes an excellent test machine because it can support both 32-bit and 64-bit versions of Windows.

  Be sure also to test your driver for uniprocessor scenarios. The easiest way to do this on a multiprocessor system is to use the /ONECPU Boot.ini option or the onecpu BCDEdit option in the system's boot settings.

• Pluggable disks on your test system, which make it easy to test against multiple versions of Windows.

• If your device supports DMA, a test machine with more than 4 GB of main memory.

  This amount might seem excessive, but it can be useful for tracking down problems in addressing extended memory. It is difficult to find such problems when your system does not have enough memory to test for them.

On the test system, you should install the checked build of the Windows version for which you are targeting your driver. The checked build is one of the most powerful tools for debugging and testing your driver. The full checked build of Windows can have a substantial impact on performance, however. As an alternative to installing the full checked build, you can build your driver in a checked build environment and install just the checked version of your driver stack. You can also replace just ntoskrnl.exe and hal.dll with the checked versions.

 Tip  See "Installing Just the Checked Operating System and HAL" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79774.

Chapter 1, "Introduction to WDF," presents a discussion about using checked and free builds of Windows for developing WDF drivers.

If you are targeting your driver for multiple versions of Windows and have a limited number of test systems, you might want to install multiple versions of Windows on each test system, simply to save time installing and reinstalling the operating system for each test pass.

Tools for Testing WDF Drivers: An Overview

Microsoft provides a number of tools for testing drivers during development. Most of these tools are available in the WDK. These tools fall into two basic categories:

• Static code analysis tools, which analyze source code for errors without actually executing the code.

  Static analysis tools include PREfast for Drivers, which can be used to test both KMDF and UMDF drivers, and Static Driver Verifier (SDV), which can be used to test KMDF drivers.

• Dynamic verification tools, which put an installed driver through its paces in hopes of activating a bug and causing the driver to fail in some way.

  Dynamic verification tools include Driver Verifier, KMDF Verifier, UMDF Verifier, and Application Verifier. These four verification tools are described in detail in this chapter.

Tracing Techniques for Testing Drivers

Tracing records the actions of a driver in a form that you can examine later by using a debugger or other tool. You can also view trace messages are they are generated. Be sure to take advantage of the following techniques in your driver.

Windows Trace Preprocessor

WPP simplifies and enhances Event Tracing for Windows, a kernel-level trace logging facility that logs both kernel and application-defined events. Generating trace messages is a relatively easy way to record what your driver is doing. You can define trace flags that correspond to the types of information you find most helpful during debugging.

Chapter 11, "Driver Tracing and Diagnosability," presents details about WPP.

The KMDF Log

KMDF includes an internal trace logger that is based on WPP. The logger tracks the progress of IRPs through the framework and the corresponding WDFREQUEST objects through the driver. It also traces Plug and Play and power state changes and other internal changes in the framework. The logger maintains a record of recent trace events for each driver instance-each KMDF driver has its own log. You can use WDF debugger extensions to view and save the KMDF log during interactive debugging.

Chapter 22, "How to Debug WDF Drivers," presents details about the KMDF logger.

About PREfast and SDV

PREfast for Drivers is a source code analysis tool that detects certain classes of errors that a compiler cannot easily find. You can use PREfast as soon as you can compile your source code-the code does not need to be linked or run.

When PREfast runs, it simulates execution of possible code paths on a function-by-function basis, including code paths that are rarely executed during runtime. PREfast checks each possible code path against a set of rules that identify potential errors or bad coding practices, and it logs warnings for code that appears to break the rules. You can use PREfast to analyze both kernel-mode drivers and other kernel-mode components.

Chapter 23, "PREfast for Drivers," provides detailed information about using PREfast to verify WDF driver source code.

SDV is a static verification tool that is designed to be run near the end of the development cycle, on drivers that have been built and are approaching the testing phase of development. SDV does deeper testing than PREfast and often finds additional errors after the errors that PREfast found have been fixed. Starting with the WDK for Windows Server Longhorn, SDV can be used to test KMDF drivers and WDM kernel-mode drivers.

SDV explores paths in the driver code by symbolically executing the source code, exercising code in paths that are missed in traditional testing. During verification, SDV examines every applicable branch of the driver code and the library code that it uses, and tries to prove that the driver violates the SDV rules. If SDV fails to prove a violation, it reports that the driver complies with the rules and passes the verification. Usage rules include general-purpose rules that apply to any kernel-mode driver and KMDF-specific rules that apply only to KMDF drivers.

Chapter 24, "Static Driver Verifier," provides details about using SDV to test WDF drivers.

Other Tools for Testing Drivers

The WDK provides many tools in addition to PREfast, SDV, and the verifiers described in this chapter. This section describes several of the more commonly used driver testing tools in the WDK. Unless otherwise noted, these tools are installed in the %wdk%\tools directory.

 Tip  See "Driver Development Tools" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79298.

INF File Syntax Checker (ChkINF)

The INF File Syntax Checker, or ChkINF, is one of the best ways to get your driver installation to work. ChkINF is a Perl script, so to use this tool you need one of the Perl interpreters identified in the WDK.

The following example shows a common approach to running ChkINF:

• ChkINF test.inf /B

This command validates Test.inf and displays the results in the default browser (/B). The report consists of a section with errors and warnings followed by an annotated copy of the INF file that shows the problems. Many of the warning messages point to problems that, in some cases, stop the driver from loading properly. You should fix all of the problems that ChkINF reports.

 Tip  ChkInf is included in the WDK at %wdk%\tools\chkinf. See "ChkINF" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79776.

Device Console (DevCon)

Device Console (Devcon.exe) is a command-line tool that displays detailed information about devices. You can use DevCon to search for and manipulate devices from the command line. DevCon enables, disables, installs, configures, and removes devices on the local computer and displays detailed information about devices on local and remote computers. DevCon runs on Windows 2000 and later versions of Windows.

You can use DevCon to verify that a driver is installed and configured correctly, including the proper INF files, driver stack, driver files, and driver package. You can also use the DevCon change commands-enable, disable, install, start, stop, and continue-in scripts that exercise the driver.

Source code for DevCon is available as a sample that demonstrates how to use the SetupAPI and device installation functions to enumerate devices and perform device operations in a console application.

 Tip  The DevCon tool is included in the WDK at %wdk%\tools\devcon; DevCon source code is at %wdk%\src\setup\devcon. See "DevCon" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79777.

Device Path Exerciser (Dc2)

Device Path Exerciser (Dc2.exe) is a command-line tool that tests the reliability and security of drivers. It calls drivers through a variety of user-mode I/O interfaces with valid, invalid, meaningless, and poorly-formatted buffers and data that will crash the driver if not managed correctly. These tests can reveal improper driver design or implementation that might result in system crashes or make the system vulnerable to malicious attacks.

Tip 

Run Device Path Exerciser while you are using Driver Verifier to monitor your driver. This produces a thorough profile of the driver's behavior during the stress of the Device Path Exerciser tests.

Device Path Exerciser identifies drivers that handle the following calls incorrectly:

• Unexpected I/O requests to the driver, such as file-system query requests directed to a sound card.

• Query requests with buffers that are too small to contain all the data to be returned.

• Device I/O control (IOCTL) requests or file-system control code (FSCTL) requests with missing buffers, buffers that are too small, or buffers that contain meaningless information.

• IOCTL or FSCTL requests with direct I/O or neither I/O data access in which the data is changing asynchronously.

• IOCTL or FSCTL requests with invalid pointers for requests that use neither I/O.

• IOCTL or FSCTL requests and fast path query requests in which the mapping of the user buffer changes asynchronously, causing the pages to become unreadable.

• Relative open operations with arbitrary or hard-to-parse file names and open operations to fictitious device objects.

During a test, Device Path Exerciser typically sends hundred of thousands of similar calls to the driver in rapid succession. These calls include varying data access methods, valid and invalid buffer lengths and addresses, and permutations of the function parameters, including spaces, strings, and characters that might be misinterpreted by a flawed parsing or error-handling routine.

 Tip  Device Path Exerciser is included in the WDK at %wdk%\tools\dc2. See "Device Path Exerciser" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79778.

KernRate and the KernRate Viewer (KRView)

The KernRate tool is primarily a profiler that helps identify where processor time is being spent. KernRate also collects statistics on a number of performance-related items in the kernel such as those in the performance monitor (Sysmon). Another tool called KRView displays KernRate output in a Microsoft Excel spreadsheet.

You should use performance profiling with your driver in two ways:

• To run a performance profile of the whole system while exercising your driver.

  Even with a significant load on your driver, the percentage of time spent in your driver should be relatively low.

• To run a performance profile on your driver alone.

  When you analyze the data from this profile, look for "hot spots" where your driver takes a lot of time. At each hot spot, look for either a problem with the code or a way to improve the algorithm in a way that reduces the load. Many hot spots represent legiti-mate activity on the part of the driver, but checking them can find bugs that affect performance.

If you have not done performance profiling before, remember that getting good data takes time. For good solid numbers, consider running a profile for hours. In addition, performance numbers are worthless unless the driver is performing common and meaningful tasks, so you need a good test that can exercise your driver for a long time. Besides time, the profile takes a lot of space-sample profilers look at the instruction pointer and put the data into a "bucket" that indicates the code being executed. KernRate allows you to adjust the size of the buckets; the smaller the bucket, the more accurate the sample.

 Info  KernRate and KRView are available for download on the WHDC Web site-online at http://go.microsoft.com/fwlink/?LinkId=79779.

Plug and Play Driver Test (Pnpdtest)

Plug and Play Driver Test (Pnpdtest.exe) exercises various Plug and Play-related code paths in the driver and user-mode components. For best results, run Plug and Play Driver Test with Driver Verifier enabled.

Plug and Play Driver Test forces a driver to handle almost all of the Plug and Play IRPs; however, it stresses three areas: removal, resource rebalance, and surprise removal. The test uses a combination of user-mode API calls through the test application and kernel-mode API calls through an upper filter driver. You can test each of these separately or test them all together as a stress test.

 Tip  Pnpdtest is included in the WDK at %wdk%\tools\pnpdtest. See "Plug and Play Driver Test" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79780.

Plug and Play CPU Test (PNPCPU)

Plug and Play CPU Test (Pnpcpu.exe) is a command-line tool that simulates adding processors to a running instance of Windows Server Longhorn. This action is sometimes called a hot add.

You can use this tool to test that your driver or application does not fail when processors are added at run time. If the driver or application has been extended to support hot add of processors on capable systems, you can use PNPCPU to confirm that all relevant Plug and Play notifications are handled correctly.

 Tip  PNPCPU is included in the WDK at %wdk%\tools\pnpcpu. See "PNPCPU" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79781.

Memory Pool Monitor (PoolMon)

Memory Pool Monitor (Poolmon.exe) displays data that the operating system collects about memory allocations from the system's paged and nonpaged memory pools and from the memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag.

You can use PoolMon to detect memory leaks when you create a new driver, change the driver code, or stress the driver. You can also use PoolMon in each stage of testing to view the driver's patterns of allocation and free operations and to reveal how much pool memory the driver is using at any given time.

PoolMon takes advantage of pool tagging to display the names of the Windows components and drivers that assign the pool tags that are listed in Pooltag.txt, a file installed with Poolmon and with Debugging Tools for Windows. To use PoolMon on Windows XP and earlier versions, you must enable pool tagging. On Windows Server 2003 and later versions of Windows, pool tagging is permanently enabled.

 Tip  PoolMon is included in the WDK at %wdk%\tools\other. See "PoolMon" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=7978

Power Management Test Tool (PwrTest)

Power Management Test Tool (PwrTest.exe) exercises and records power management information from the system. You can use PwrTest to automate sleep and wake transitions and to record processor power management and battery information from the system over a period of time.

 Tip  PwrTest is included in the WDK at %wdk%\tools\acpi\pwrtest. See "PwrTest" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79783.

PwrTest features robust logging and a command-line interface. It tests several power management scenarios, as described in Table 21-1.

Table 21-1: Power Management Scenarios Tested by PwrTest
Open table as spreadsheet

Scenario	What is tested
SLEEP	Exercises sleep and resume transition functionality. Sleep/resume transitions can be automated, and target sleep states can be specified.
PPM	Displays and records processor power management (PPM) information and metrics. Performance (ACPI P state) and processor idle (ACPI C state) information can be logged for a period of time at a specified interval.
BATTERY	Displays and records battery information and metrics. Battery capacity, voltage, rate of drain, and estimated remaining lifetime can be logged for a period of time at a specified interval.
INFO	Displays system power management information, such as available sleep states and processor power management capabilities.
ES	Displays and records thread execution state changes. Changing thread execution states enables applications and services to temporarily override system power management settings such as the monitor timeout and sleep idle timeout.

Windows Device Testing Framework

The Microsoft Windows Device Testing Framework (WDTF) exposes a set of COM interfaces that you can use to write your own device-centric, scenario-based automated tests. WDTF simplifies several basic device testing scenarios:

• Finding devices that are attached to the computer.

  You can search for devices by technology such as USB or Bluetooth, by device class such as a storage device, or by other criteria. You can also identify a particular device by name or use multiple criteria to search for devices. In addition, you can search for devices that are related to each other by parent-child or graph relationships.

• Creating collections of testable targets.

  This functionality is useful when you want to test devices that have common characteristics. For example, you can query for the collection of all disk devices and then test the collection as a group.

• Performing actions on one or more different devices.

  You can build a generic scenario to test a set of devices even when the individual devices are of different types.

• Running simple functionality tests on one or more devices without knowing the underlying type of device.

  You can perform simple synchronous I/O operations on a device by using the SimpleIO device action interface. SimpleIO abstracts I/O operations so you can use a standard interface to initiate the test on each device, even though the underlying code that performs the actions is different for different devices. The framework has a library of SimpleIO implementations and identifies and instantiates the correct one for each device type on your behalf.

• Programmatically start, stop, or pause stress tests.

  You can perform simple I/O operations on a device asynchronously and control the test at runtime by using the SimpleIOStress interface. SimpleIOStress is implemented as a wrapper around SimpleIO.

• Use test implementation code without specific device expertise.

  You can write WDTF scripts that use code that is specific to devices and device classes without understanding how each device implements the interfaces that you use.

You can write WDTF test scripts in any programming language that supports COM automation. The WDK provides the following sample scripts, which you can use to test KMDF and UMDF drivers:

This script …	Performs this test …
Disable_Enable_With_IO.wsf	Disables and enables each device that can be disabled, one by one. This sample also runs I/O on all of the devices that are related to the selected devices.
Sleep_Stress_With_IO.wsf	Exposes devices to power management stress testing.
Common_Scenario_Stress_With_IO.wsf	Exposes devices to power management and Plug and Play stress testing.
Basic_SimpleIO.wsf	Performs the "With_IO" part of the standard scenarios that are described for the preceding scripts.
EnumDevices.wsf	Displays information about all devices that are found on the computer.

 Tip  WDTF libraries and sample scenarios are provided in the WDK at %wdk%\tools\WDTF. See "Windows Device Testing Framework" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79785.

Tip 

WDTF is automatically installed by Driver Test Manager (DTM), a test harness provided in the WDK for running tests for submission to the Windows Logo Program. If you use DTM, you are not required to install WDTF as a separate step.