Driver Verifier
Driver Verifier is an extremely useful tool for detecting errors in kernel-mode drivers. It tests and traps many conditions that might otherwise go unnoticed in normal operation. Driver Verifier verifies that drivers are not making illegal function calls or causing system corruption. Driver Verifier is installed in all versions of Windows Vista, Windows Server 2003, Windows XP, and Windows 2000.
| Tip | All kernel-mode drivers that are developed internally at Microsoft must pass Driver Verifier with standard settings. Microsoft developers recommend that you enable Driver Verifier as early as possible and use it throughout the development of your driver. |
Driver Verifier has evolved over time, so versions of Driver Verifier installed in later releases of Windows have more features than earlier versions. A significant enhancement in Driver Verifier for Windows Vista is the ability to activate most options and to add and remove drivers for verification without rebooting the computer.
Depending on the options that are enabled and the number of drivers being verified, running Driver Verifier can affect performance on the test system. When you are testing your driver for performance-related issues, be sure to disable Driver Verifier so it does not skew the results.
Note Driver Verifier validates the use of the kernel-mode WDM function calls. When you run Driver Verifier on your WDF driver, the output might contain the names of functions that your driver does not call explicitly but that the framework calls on your driver's behalf. The discussion in this section lists the underlying WDM function names.
Tip See "Driver Verifier" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79793. See also "Driver Verifier in Windows Vista" on the WHDC Web site-online at http://go.microsoft.com/fwlink/?LinkId=79588.
When to Use Driver Verifier
You can run Driver Verifier throughout the development and test process to help find problems early in the development life cycle, when they are easier and cheaper to correct. Driver Verifier can identify conditions such as memory corruption, mishandled IRPs, invalid DMA buffer usage, and possible deadlocks. Because Driver Verifier is installed with Windows, it can also be used to troubleshoot problems on customers' computers.
How Driver Verifier Works
Driver Verifier operates on drivers that are installed and running on the system. Driver Verifier works by intercepting certain system calls from the driver and injecting faults or constraining resources, to test the driver's behavior under extreme conditions.
When Driver Verifier runs, it always checks whether the driver uses memory at an improper IRQL, improperly acquires or releases spin locks, improperly allocates or frees memory, or frees pool memory allocations without first removing timers. When the driver is unloaded, Driver Verifier always checks to see that it has properly released its resources.
Table 21-2 lists options that you can specify to direct Driver Verifier to perform additional checks.
| When this option is active… | Driver Verifier… |
|---|---|
| Deadlock Detection | Monitors the driver's use of spin locks, mutexes, and fast mutexes to detect whether the driver's code has the potential to cause a deadlock. (Windows XP and later) |
| Disk Integrity Checking | Monitors hard disk access and detects whether the disk is preserving its data correctly. (Windows Server 2003 and later) |
| DMA Verification | Monitors the driver's use of DMA routines to detect improper use of DMA buffers, adapters, and map registers. (Windows XP and later) |
| Driver Hang Verification | Times the driver's I/O completion and cancellation routines and reports routines that exceed time limits that you specify. (Windows Vista and later) |
| Enhanced I/O Verification | Monitors the calls of several I/O manager routines and performs stress testing of Plug and Play IRPs, power IRPs, and WMI IRPs. (Windows XP and later) |
| Force IRQL Checking | Places extreme pressure on the driver by invalidating pageable code. If the driver attempts to access paged memory at the wrong IRQL or while holding a spin lock, Driver Verifier detects this behavior. |
| Force Pending I/O Requests | Tests the driver's response to STATUS_PENDING return values by returning STATUS_PENDING for random calls to IoCallDriver. (Windows Vista and later) |
| I/O Verification | Allocates the driver's IRPs from a special pool and monitors the driver's I/O handling to detect illegal or inconsistent use of I/O routines. In Windows Vista and later versions of Windows, activating I/O Verification also activates Driver Hang Verification. |
| IRP Logging | Monitors a driver's use of IRPs and creates a log of IRP use. (Windows Server 2003 and later) |
| Low Resources Simulation | Randomly fails pool memory allocation requests and other resource requests. By injecting these allocation faults into the system, Driver Verifier tests the driver's ability to cope with a low-resource situation. |
| Miscellaneous Checks | Looks for common causes of driver crashes, such as the mishandling of freed memory. (Windows Vista and later) |
| Pool Tracking | Checks whether the driver has freed all of its memory allocations when it is unloaded, revealing memory leaks. |
| Special Pool | Allocates most of the driver's memory requests from a special pool that is monitored for memory overruns, memory underruns, and memory that is accessed after it is freed. |
Open table as spreadsheet
How to Run Drivser Verifier
When you start Driver Verifier, you specify the driver or drivers to verify and the options to apply. Driver Verifier then monitors the behavior of the selected drivers until you either deactivate Driver Verifier or reboot the system. You can direct Driver Verifier output to a log file or use the !verifier debugger extension to view Driver Verifier in a debugger, if one is attached to the system.
| Inside Out | The Verifier utility (Verifier.exe) activates and monitors Driver Verifier. The Verifier utility is located in the %windir%\system directory. |
To Use the Verifier Utility as a Command-Line Tool
-
Enter a verifier command with at least one command-line parameter.
For example, the following command line activates Driver Verifier with pool tracking (/flags 8) for the driver Xyz.sys:
-
verifier /flags 8 /driver xyz.sys
Tip See "The Verifier Command Line" in the WDK for complete details about Verifier command-line options-online at http://go.microsoft.com/fwlink/?LinkId=79788.
To Run Driver Verifier Manager
-
Type verifier with no command-line parameters.
The Verifier utility starts Driver Verifier Manager, which provides a simple graphical user interface you can use to select settings and drivers to verify.
In subsequent screens, you can select predefined or individual settings and the drivers to verify. Driver Verifier Manager then launches the Verifier utility with those settings enabled.
Tip See "Driver Verifier Manager (Windows XP and Later)" in the WDK-online at http://go.microsoft.com/fwlink/?LinkId=79789.
Driver Verifier Examples
The following examples show Verifier commands that use a few typical parameters. In practice, you would combine parameters to activate the checks that make sense for your driver and testing scenarios.
Example 1: Activate Standard Options for a List of Drivers
The following command line activates a set of "standard" Driver Verifier options for the specified drivers after the next boot:
-
verifier /standard /drivers driverlist
The /standard parameter in Windows XP and later versions activates the following Driver Verifier options:
| Deadlock Detection | I/O Verification |
| DMA Verification | Pool Tracking |
| Force IRQL Checking | Special Pool |
In Windows Vista and later versions, the /standard options also include Security Checks and Miscellaneous Checks.
The /drivers parameter directs Driver Verifier to verify the specified drivers. Driverlist is a list of drivers by binary name, such as Driver.sys. Use a space between driver names.
Example 2: Activate Specific Options for All Drivers
The following command line sets one or more options for all drivers on the system:
-
verifier /flags options /all
The /flags parameter specifies one or more options to activate after the next reboot. Options can be specified in either decimal or hexadecimal format in any combination of the values in Table 21-3.
| Decimal | Hexadecimal | Option |
|---|---|---|
| 1 | 0x1 | Special Pool |
| 2 | 0x2 | Force IRQL Checking |
| 4 | 0x4 | Low Resources Simulation |
| 8 | 0x8 | Pool Tracking |
| 16 | 0x10 | I/O Verification |
| 32 | 0x20 | Deadlock Detection (Windows XP and later) |
| 64 | 0x40 | Enhanced I/O Verification (Windows XP and later) |
| 128 | 0x80 | DMA Verification (Windows XP and later) |
| 256 | 0x100 | Security Checks (Windows XP and later) |
| 512 | 0x200 | Force Pending I/O Requests (Windows Vista and later) |
| 1024 | 0x400 | IRP Logging (Windows Server 2003 and later) |
| 2048 | 0x800 | Miscellaneous Checks (Windows Vista and later) |
For example, in Windows Vista the /standard parameter shown in Example 1 is the equivalent of /flags 0x9BB.
Example 3: Start or Stop the Verification of a Driver without Rebooting
To start the verification of any driver without rebooting the system, even when the driver is already loaded, use the following command syntax:
-
verifier /volatile /adddriver DriverName.sys
The /volatile parameter changes settings without rebooting the computer. Volatile settings take effect immediately.
To remove a driver from the list of drivers to verify, use the /removedriver parameter, as shown in the following example:
-
verifier /volatile /removedriver DriverName.sys
The /removedriver parameter removes the driver from the verification list only if the driver is not already loaded. If the driver is already loaded, Driver Verifier continues to monitor that driver until you reboot the system. To minimize overhead until the next reboot, deactivate all Driver Verifier options as shown in Example 5.
Example 4: Activate or Deactivate Options without Rebooting
To enable and disable any option without rebooting, use /volatile with the /flags parameter, as shown in the following example:
-
verifier /volatile /flags [options]
You can use this command syntax with any Driver Verifier option except SCSI Verification and Disk Integrity Checking. For example, the following command activates the Deadlock Detection option without rebooting:
-
verifier /volatile /flags 0x20
Example 5: Deactivate All Driver Verifier Options
To deactivate Driver Verifier options without rebooting, use /flags 0 together with /volatile, as shown in the following example:
-
verifier /volatile /flags 0
Driver Verifier continues to monitor the driver by using the options in the automatic checks feature, which cannot be turned off. However, the overhead of automatic checks is approximately 10 percent of the overhead of a typical verification, so it has relatively little impact on performance.
Example 6: Deactivate Driver Verifier
To clear all Driver Verifier settings, use the following command line:
-
verifier /reset
After the next reboot, no drivers will be verified.
Example 7: Use Low Resources Simulation
When the Low Resources Simulation option is active, Driver Verifier fails random instances of the driver's memory allocations as might occur if the driver is running on a computer with insufficient memory. This tests the driver's ability to respond properly to low-memory and other low-resource conditions.
The Low Resources Simulation test fails allocations that are requested by calls to several different functions, including the following:
| ExAllocatePoolXxx | MmMapIoSpace |
| MmMapLockedPagesSpecifyCache | MmProbeAndLockPages |
In Windows Vista and later versions, calls to the following functions are also injected with the following faults:
| IoAllocateErrorLogEntry | IoAllocateWorkItem |
| IoAllocateIrp | MmAllocateContiguousMemoryXxx |
| IoAllocateMdl | MmAllocatePagesForMdl |
To enable Low Resources Simulation, use the following command:
-
verifier /faults [Probability PoolTags Applications DelayMins] /driver DriverList
The custom settings parameters-Probability, PoolTags, and so on-must appear in the order displayed. If you omit a value, type quotation marks to hold its place.
The following parameters are used for Low Resources Simulation:
-
Probability
Specifies the probability that Driver Verifier will fail a given allocation. The probability is specified as a number-in decimal or hexadecimal format-of chances in 10,000 that Driver Verifier will fail the allocation. The default value-600-means 600/10000, or 6 percent.
-
PoolTags
Specifies the pool tags for which Driver Verifier can fail memory allocations. By default, all allocations can fail. You can use a wildcard character (*) to represent multiple pool tags-for example, abc*. To list multiple pool tags, separate the tags with spaces and enclose the list in quotation marks.
-
Applications
Specifies the programs for which Driver Verifier can fail allocations. Applications consists of the name of one or more executable files. By default, allocations in all applications can fail. To list programs, separate the program names with spaces and enclose the list in quotation marks.
-
DelayMins
Specifies the number of minutes after booting during which Driver Verifier does not intentionally fail any allocations. This delay allows the drivers to load and the system to stabilize before the test begins. DelayMins is specified as a number in decimal or hexadecimal format. The default value is 7.
The following command enables Low Resources Simulation with a probability of 10 percent (1000/10000) and a delay of 5 minutes for the pool tags named Tag1 and Fred plus the application Notepad.exe:
-
verifier /faults 1000 "Tag1 Fred" Notepad.exe 5
The following command enables Low Resources Simulation with the default values, except that it extends the delay to 10 minutes:
-
verifier /faults "" "" ""0xa
You can use the /volatile command-line parameter to change Low Resources Simulation settings without rebooting the computer, as in the following command:
-
verifier /volatile /faults [Probability PoolTags Applications DelayMins /driver Driverlist]
Volatile settings take effect immediately.
Example 8: Use Force IRQL Checking
Although kernel-mode drivers cannot access pageable memory at a high IRQL or while holding a spin lock, such an action might not trigger an error if the memory page has not actually been removed from the working set and paged out to disk.
When the Force IRQL Checking option is enabled, Driver Verifier trims all of the driver's pageable code and data-including pageable pool, code, and data-from the working set whenever the selected driver attempts to acquire a spin lock, calls KeSynchronizeExecution, or raises the IRQL to DISPATCH_LEVEL or higher. If the driver attempts to access any of this memory, Driver Verifier issues a bug check.
In Windows Vista, Driver Verifier can detect when certain synchronization objects are allocated in pageable memory. These synchronization objects should not be allocated from pageable memory because the kernel might access them at elevated IRQL. Driver Verifier can detect the following pageable objects:
| ERESOURCE | KSEMAPHORE |
| FAST_MUTEX | KSPIN_LOCK |
| KEVENT | KTIMER |
| KMUTEX |
Force IRQL Checking is included in the /standard settings. To enable Force IRQL Checking individually, use the following command:
-
verifier /flags 0x2 /driver DriverList
The feature is active after the next boot. Use the /volatile parameter to activate or deactivate Force IRQL Checking without rebooting.
How to Use Driver Verifier Information during Debugging
The !verifier debugger extension can be used to monitor and report statistics related to Driver Verifier during a debugging session. The example in this section shows how to use !verifier to debug a crash caused by Driver Verifier's Low Resources Simulation option.
This information applies for any version of Windows when Driver Verifier is using Low Resources Simulation.
Example 1: Use !verifier to View Stack Traces
The easiest crashes to understand are probably those where a driver is accessing a NULL pointer. For this example, inspecting the source code around the path that crashed reveals that the driver called ExAllocatePoolWithTag, the function returned NULL, and the driver did not check the return value, so it crashed when using the pointer.
Understanding the cause of a driver crash is not always trivial, however. Often you can extract useful information by looking at the stack traces for recently injected faults. For example, !verifier 4 in the kernel debugger displays the four most recent injected faults-not necessarily four stack traces. You can specify an additional parameter for !verifier to display more stack traces. The stack trace for the most recent fault appears first.
If you encounter a crash dump or a debugger break while fault injection is enabled, you can use the stack trace to see if fault injection was the root cause. For example, if the debugger indicated a crash in win32k!GreEnableEUDC, Listing 21-1 would tell you that the crash was the result of the last induced fault.
Listing 21-1: Example of crash in win32k!GreEnableEUDC
 |
kd> !verifier 4 Resource fault injection history: Tracker @ 8354A000 (# entries: 80, size: 80, depth: 8) Entry @ 8354B258 (index 75) Thread: C2638220 816760CB nt!VerifierExAllocatePoolWithTag+0x49 A4720443 win32k!bDeleteAllFlEntry+0x15d A4720AB0 win32k!GreEnableEUDC+0x70 A47218FA win32k!CleanUpEUDC+0x37 A473998E win32k!GdiMultiUserFontCleanup+0x5 815AEACC nt!MiDereferenceSession+0x74 8146D3B4 nt!MmCleanProcessAddressSpace+0x112 815DF739 nt!PspExitThread+0x603 Entry @ 8354B230 (index 74) Thread: 8436D770 816760CB nt!VerifierExAllocatePoolWithTag+0x49 A462141C win32k!Win32AllocPool+0x13 A4725F94 win32k!StubGdiAlloc+0x10 A4631A93 win32k!ExAllocateFromPagedLookasideList+0x27 A47261A4 win32k!AllocateObject+0x23 A4726F76 win32k!HmgAlloc+0x25 A47509D8 win32k!DCMEMOBJ::DCMEMOBJ+0x3b A4717D61 win32k!GreCreateDisplayDC+0x31 Entry @ 8354B208 (index 73) Thread: D6B4B9B8 816760CB nt!VerifierExAllocatePoolWithTag+0x49 A462141C win32k!Win32AllocPool+0x13 A46C2759 win32k!PALLOCMEM+0x17 A477CCF2 win32k!bComputeGISET+0x82 A477D07D win32k!PFEMEMOBJ::bInit+0x248 A475DC18 win32k!PFFMEMOBJ::bAddEntry+0x6c A475E3E6 win32k!PFFMEMOBJ::bLoadFontFileTable+0x81 A475BADE win32k!PUBLIC_PFTOBJ::bLoadFonts+0x2c4 Entry @ 8354B1E0 (index 72) Thread: CCA0A480 816760CB nt!VerifierExAllocatePoolWithTag+0x49 813B8C30 fltmgr!FltpAllocateIrpCtrl+0x122 813CB1C9 fltmgr!FltpCreate+0x28d 81675275 nt!IovCallDriver+0x1b1 8141EDF1 nt!IofCallDriver+0x1f 81566106 nt!IopParseDevice+0xde6 815B9916 nt!ObpLookupObjectName+0x61a 815B72D5 nt!ObOpenObjectByName+0xf7
|
The most recent allocation failure was induced on the GreEnableEUDC code path. Remember that GreEnableEUDC was the function that crashed in this example. Note that the allocation failure occurred in the context of thread C2638220. If you run !thread -1 and the address of the current thread is C2638220, then it is even more probable that the most recent fault was related to the current crash. You should review the source code around that area, looking for a code path that could cause that kind of crash.
Often, the current crash is related to the most recently injected failure. If looking at the most recent stack trace is not helpful, look at other stack traces. If these are not helpful either, you can use !verifier 4 80 to display the most recent 0x80 stack traces; one of these might prove to be useful.
Example 2: Use !verifier to Display Fault and Pool Allocation Counters
Driver Verifier keeps track of the number of faults injected since the last reboot. Also, Driver Verifier keeps track of the number of attempted pool allocations. These two numbers can help you understand the following:
-
Whether Low Resources Simulation is actually injecting faults in the driver being tested.
For example, if the driver is not allocating pool memory, Low Resources Simulation does not inject faults.
-
Whether an excessive number of faults were injected.
For example, if the number of reported injected faults is too large when compared to the number of attempted allocations, you can adjust the fault injection probability to a smaller value for the next test pass.
-
Whether too few faults were injected.
In this case, you should increase the fault injection probability for future test passes.
Driver Verifier's pool allocation counters can be displayed by using !verifier as shown in Listing 21-2.
Listing 21-2: Example display of pool allocation counters
|
!verifier Verify Level 5 ... enabled options are: Special pool Inject random low-resource API failures Summary of All Verifier Statistics RaiseIrqls 0x2c671f AcquireSpinLocks 0xca1a02 Synch Executions 0x10a623 Trims 0x0 Pool Allocations Attempted 0x862e0e Pool Allocations Succeeded 0x8626e3 Pool Allocations Succeeded SpecialPool 0x768060 Pool Allocations With NO TAG 0x0 Pool Allocations Failed 0x34f Resource Allocations Failed Deliberately 0x3f5
|
EAN: 2147483647
Pages: 224