Types of Windows Debuggers

[Previous] [Next]

If you've been programming Windows for any length of time, you've probably heard about several different types of debuggers that you can use. Two types of debuggers are available in the Windows world: user-mode debuggers and kernel-mode debuggers.

User-mode debuggers are much more familiar to most developers. Not surprisingly, user-mode debuggers are for debugging user-mode applications. The Microsoft Visual C++ debugger is a prime example of a user-mode debugger. Kernel-mode debuggers, as the names implies, are those that let you debug the operating system kernel. These debuggers are used mostly by device driver writers when they're debugging their device drivers.

User-Mode Debuggers

User-mode debuggers are used to debug any application that runs in user mode, which includes any GUI programs as well as applications you wouldn't expect, such as Windows 2000 services. Generally, user-mode debuggers use GUIs. The main hallmark of user-mode debuggers is that they use the Win32 Debugging application programming interface (API). Because the operating system marks the debuggee as running in a special mode, you can use the IsDebuggerPresent API function to find out whether your process is running under a debugger.

The Win32 Debugging API comes with an implied contract: once a process is running under the Debugging API, thus making it a debuggee, the debugger can't detach from that process. This symbiotic relationship means that if the debugger ends, the debuggee will end as well. The debugger is also limited to debugging only the debuggee and any processes spawned by the debuggee (if the debugger supports descendant processes).

For interpreted languages and run times that use a virtual machine approach, the virtual machines themselves provide the complete debugging environment and don't use the Win32 Debugging API. Some examples of those types of environments are the Microsoft or Sun Java Virtual Machines (VMs), the Microsoft scripting environment for Web applications, and the Microsoft Visual Basic p-code interpreter.

We'll get to Visual Basic debugging in Chapter 7, but be aware that the Visual Basic p-code interface is undocumented. I won't go into the Java and scripting debugging interfaces, subjects beyond the scope of this book. For more information on debugging and profiling the Microsoft Java VMs, search for the "Debugging and Profiling Java Applications" topic on MSDN. The interfaces are rich and extensive and allow you to completely control the operation of the Java VM. For information on writing a script debugger, search MSDN for the topic, "Active Script Debugging API Objects." Like the Java VM, the script debugger objects provide a rich interface for accessing scripts and document-hosted scripts.

A surprising number of programs use the Win32 Debugging API. These include the Visual C++ debugger, which I cover in depth in Chapters 5 and 6; the Windows Debugger (WinDBG), which I discuss in the kernel-mode debugger section that follows; Compuware NuMega's BoundsChecker; the Platform SDK HeapWalker program; the Platform SDK Depends program; the Borland Delphi and C++ Builder debuggers; and the NT Symbolic Debugger (NTSD). I'm sure there are many more.

Kernel-Mode Debuggers

Kernel-mode debuggers sit between the CPU and the operating system. That means that when you stop in a kernel-mode debugger, the operating system also stops completely. As you can imagine, bringing the operating system to an abrupt halt is helpful when you're working on timing and synchronization problems. Except for one kernel-mode debugger that I'll talk about later (in the section "SoftICE"), however, you can't debug user-mode code with kernel-mode debuggers.

There aren't that many kernel-mode debuggers. A few are the Windows 80386 Debugger (WDEB386), the Kernel Debugger (i386KD), WinDBG, and SoftICE. I'll briefly describe each of these debuggers in the following sections.

WDEB386

WDEB386 is the Windows 98 kernel-mode debugger distributed with the Platform SDK. This debugger is useful only for developers writing Windows 98 virtual device drivers (VxDs). Like most kernel-mode debuggers for Windows operating systems, WDEB386 requires two machines and a null modem cable to work. Two machines are necessary because the kernel-mode portion that runs on the target machine has limited access to the target machine's hardware, so it sends its output to and receives its commands from another machine.

WDEB386 has an interesting history. It started out as a Microsoft internal tool back in the Windows 3.0 days. It's hard to use and doesn't have much support for source-level debugging and other nice features that the Visual C++ and Visual Basic debuggers have spoiled us with.

The "DOT" commands are the most important feature of WDEB386. Through INT 41, you can extend WDEB386 to add more commands to the debugger. This extensibility allows VxD writers to create custom debugger commands that give them easy access to information in their VxDs. The Windows 98 debug version supports a plethora of DOT commands that allow you to observe the exact state of the operating system at any point in the debugging process.

i386KD

Windows 2000 is different from Windows 98 in that the actual kernel-mode debugger portion is part of NTOSKRNL.EXE, the main kernel file of the Windows 2000 operating system. This debugger is available in both the free (release) and checked (debug) builds of the operating system. To turn on kernel-mode debugging, set the /DEBUG boot option in BOOT.INI and, additionally, the /DEBUGPORT boot option if you need to set the communications port for the kernel-mode debugger to a port other than the default (COM1). i386KD runs on its own machine and communicates with the Windows 2000 machine through a null modem cable.

The NTOSKRNL.EXE kernel-mode debugger does just enough to control the CPU so that the operating system can be debugged. The bulk of the debugging work—handling symbols, advanced breakpoints, and disassembly—happens on the i386KD side. At one time, the Windows NT 4 Device Driver Kit (DDK) documented the protocol used across the null modem cable. However, Microsoft no longer documents this protocol.

The power of i386KD is apparent when you see all the commands that it offers for accessing the internal Windows 2000 state. If you've ever wanted to see what happens in the operating system, these commands will show you. Having a working knowledge of how Windows 2000 device drivers work will help you follow much of the command's output. For all its power, i386KD is almost never used because it's a console application, which makes it tedious to use with source-level debugging.

WinDBG

WinDBG is the debugger that comes with the Platform SDK. You can also download it from http://msdn.microsoft.com/developer/sdk/default.asp . It's a hybrid debugger in that it can be a kernel-mode debugger as well as a user-mode debugger, but WinDBG won't let you debug both kernel-mode and user-mode programs at the same time. For kernel-mode debugging, WinDBG offers all the same power of i386KD but in an easier to use GUI front end that supports much simpler source-level debugging. With WinDBG, you can debug your device drivers nearly as easily as you would your user-mode applications.

As a user-mode debugger, WinDBG is good, and I strongly recommend that you install it if you haven't already. WinDBG offers more power than the Visual C++ debugger in that WinDBG shows you much more information about your process. However, that power does come at a cost: WinDBG is harder to use than the Visual C++ debugger. Still, I'd advise you to spend some time and effort learning about WinDBG. The investment might pay off by helping you solve a bug much faster than you can with the Visual C++ debugger alone. On average, I find that I spend about 70 percent of my debugging time in the Visual C++ debugger and the rest in WinDBG.

The first time you start WinDBG, you'll see that it has a Command window. Like the Visual C++ debugger, WinDBG does source-level debugging. However, the real power of WinDBG is found in its Command window interface. If you get comfortable with the different commands, you might find that you can debug faster with the Command window than you can with just a GUI.

The importance of the Command window increases when you add your own commands, called WinDBG extensions, to WinDBG. Whereas the Visual C++ debugger is only a little flexible when your process is stopped in the debugger, WinDBG has an entire API that allows you to get at all the debugger functionality, including the disassembler, the symbol engine, and the stack trace engine. For more information about WinDBG extensions, search for the "Debugger Extension" topic on MSDN.

In certain situations, I use WinDBG rather than the Visual C++ debugger because WinDBG supports more powerful breakpoints. Because of the Command window, you can associate commands with a breakpoint. If you stop and think about this capability, you'll realize that it allows you to take your debugging to new levels. For example, when you're tracking down a problem in a module you call many times, it would be nice if you could see the values at each call without stopping the application. With WinDBG, you can create a breakpoint command that dumps the data and then resumes execution of the program. In the Command window, you'll have a stream of all the data values leading up to the problem.

In addition to providing better debugger extensibility than the Visual C++ debugger does, WinDBG has one feature that you should definitely consider if your application runs on Windows 2000 or Windows NT 4: WinDBG can read user-mode dump files created by Dr. Watson. This capability means that you can load the exact state of your program at the time of the crash into the debugger so that you can see exactly where the program was when it crashed. See my December 1999 and January 2000 "Bugslayer" columns in Microsoft Systems Journal for more information about what you need to get this feature set up and working on your systems.

WinDBG will never become my full-time debugger because the Visual C++ debugger is so much easier to use. However, I hope that I've piqued your interest in using it. The power and capabilities it offers can help you find some very nasty problems more easily than you can with the Visual C++ debugger alone. A small investment in learning WinDBG can mean saving a huge amount of time when it comes to figuring out that one tough crash.

SoftICE

SoftICE is a commercial kernel-mode debugger from Compuware NuMega and is the only commercial kernel-mode debugger (that I know of) on the market. It's also the only kernel-mode debugger that can operate on a single machine. Unlike the other kernel-mode debuggers, however, SoftICE does an excellent job debugging user-mode programs. As I mentioned earlier, kernel-mode debuggers sit between the CPU and the operating system; SoftICE also sits between the CPU and the operating system when debugging a user-mode program, thereby stopping the entire operating system dead.

At first glance, you might not be impressed by the fact that SoftICE can bring the operating system to a halt. But consider this question: What happens if you have some timing-dependent code you need to debug? If you're using an API function such as SendMessageTimeout, you can easily time out as you step through another thread with a typical GUI debugger. With SoftICE, you can step all you want because the timer that SendMessageTimeout relies on won't be executing while you're running under SoftICE. SoftICE is the only debugger that allows you to effectively debug multithreaded applications. The fact that SoftICE stops the entire operating system when it's active means that solving timing problems is far easier.

Another benefit of SoftICE sitting between the CPU and the operating system is that debugging cross-process interactions becomes very easy. If you're doing COM programming with multiple out-of-process servers, you can easily set breakpoints in all the processes and step between them. Finally, if you do need to step from user mode to kernel mode and back, SoftICE makes such shifting trivial.

SoftICE really shines when you need to stop the application when it's accessing a particular piece of memory. SoftICE takes advantage of the i386 debug registers, which allow you to specify up to a 4-byte piece of memory to break on. You can break when the memory location is read from, written to, or executed. Because SoftICE can use up to four hardware breakpoints, your application runs at full speed until your memory location is accessed. This feature is invaluable for tracking down memory-corruption problems.

The other major advantage that SoftICE has over all other debuggers is that it has a phenomenal collection of informational commands that let you see virtually everything that's happening in the operating system. Although i386KD and WinDBG have a substantial number of these commands, SoftICE has many more. You can view almost anything in SoftICE, from the state of all synchronization events, to complete HWND information, to extended information about any thread in the system. I run SoftICE all the time and can't imagine developing software without it. It's one of the best tools I've ever used. In fact, on Windows 98 I don't use any other debuggers.

As you might expect, all this wonderful raw power has a price tag. SoftICE, like any kernel-mode debugger, has a steep learning curve because it's essentially its own operating system when it's running. However, your return on investment makes learning how to use SoftICE worth the effort.

Common Debugging Question

---

How do I change the default debugger that the operating system will use when a crash occurs?

When an application crashes, Windows 2000 looks in the registry key HKEY- _LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug , and Windows 98 looks in the [AeDebug] section of WIN.INI to determine what they should call to debug the application. If no values are in the key, Windows 2000 reports the address of the crash. If an access violation caused the crash, Windows 2000 also reports the memory location that the process couldn't read or write. Windows 98 displays the standard crash dialog box, and if you click the Details button, it will list the module, address, and registers at the time of the crash.

Three possible string values can be placed in the AeDebug key or section.

• Auto
• Debugger
• UserDebuggerHotKey

If Auto is set to 0 (zero), the operating system will generate the standard crash dialog box and enable the Cancel (Windows 2000) or Debug (Windows 98) button if you want to attach the debugger. If Auto is set to 1 (one), the debugger is automatically started. The Debugger value specifies the debugger the operating system will start on the crashed application. The only requirement for the debugger is that it supports attaching to a process. The UserDebuggerHotKey value identifies the key that will be used to break into the debugger. Refer to the section "Quick Break Keys" later in the chapter to find out how to set this value.

You can set the AeDebug key manually, but Dr. Watson (Windows 2000 only), WinDBG, and the Visual C++ debugger allow you to set it through various means. Dr. Watson and WinDBG use the -I command-line switch that will set them as the default debugger. To set the Visual C++ debugger as the debugger the operating system will call, on the Debug tab in the Options dialog box, check Just-In-Time Debugging.

If you do look at the AeDebug key, the value that's entered for Debugger looks like a string passed to the wsprintf API function: "drwtsn32 -p %ld -e %ld -g." That's exactly what it is. The -p is the process ID for the crashing process, and the -e is an event handle value that the debugger needs to signal when its debug loop gets the first thread exit debug event. Signaling the event handle tells the operating system that the debugger attached cleanly.