Types of Windows Debuggers

If you've been programming Win32 for any length of time, you've probably heard about several different types of debuggers that you can use. Two types of debuggers are available in the Microsoft Windows world: user-mode debuggers and kernel-mode debuggers.

User-mode debuggers are much more familiar to most developers. Not surprisingly, user-mode debuggers are for debugging user-mode applications. The Microsoft Visual Studio .NET debugger is a prime example of a user-mode debugger. Kernel-mode debuggers, as the names implies, let you debug the operating system kernel. These debuggers are used mostly by device driver writers when they're debugging their device drivers.

User-Mode Debuggers

As I just mentioned, user-mode debuggers are used to debug any application that runs in user mode, which includes any GUI programs as well as applications you wouldn't expect, such as Windows services. Generally, user-mode debuggers use GUIs. The main hallmark of user-mode debuggers is that they use the Win32 Debugging API. Because the operating system marks the debuggee as running in a special mode, you can use the IsDebuggerPresent API function to find out whether your process is running under a debugger. Checking whether you're running under a debugger can be useful when you want to turn on more diagnostic information only when a debugger is attached to your process.

On Microsoft Windows 2000 and earlier operating systems, the problem with the Win32 Debugging API was that once a process was running under the Debugging API, if the debugger terminated, the debuggee terminated as well. In other words, the debuggee was permanently debugged. This limitation was fine when everyone was working on client applications, but it was the bane of server application development, especially when programmers were trying to debug production servers. With Microsoft Windows XP, Windows Server 2003, and later, you can attach and detach all you want from running processes without any special steps. With Visual Studio .NET, you can detach from a process by selecting it in the Processes dialog and selecting Detach.

Interestingly, Visual Studio .NET now offers the Visual Studio Debugger Proxy (DbgProxy) service on Windows 2000, which allows you to debug processes that you can detach from. DbgProxy serves as the debugger, so your application is still running under a debugger. Now, however, you can detach and re-attach all you want under Windows 2000. One problem I keep seeing with developers—regardless of whether we're using Windows XP, Windows Server 2003, or DbgProxy on Windows 2000—is that because we're so used to the old "debugger forever" problem, we forget to take advantage of the new ability to detach.

For interpreted languages and run times that use a virtual machine approach, the virtual machines themselves provide the complete debugging environment and don't use the Win32 Debugging API. Some examples of those types of environments are the Microsoft or Sun Java Virtual Machines (VMs), the Microsoft scripting environment for Web applications, and, of course, the Microsoft .NET common language runtime (CLR).

As I mentioned earlier, .NET CLR debugging is fully covered by the documents in the Tool Developers Guide directory. I also won't go into the Java and scripting debugging interfaces, subjects beyond the scope of this book. For information on writing a script debugger, search MSDN for the "Microsoft Windows Script Interfaces-Introduction" topic. Like .NET CLR debugging, the script debugger objects provide a rich interface for accessing scripts and document-hosted scripts.

A surprising number of programs use the Win32 Debugging API. These include the Visual Studio .NET debugger when debugging native code, which I cover in depth in Chapter 5 and Chapter 7; the Windows Debugger (WinDBG), which I discuss in Chapter 8; Compuware NuMega's BoundsChecker; the Platform SDK Depends program (which you can install as part of Visual Studio .NET); the Borland Delphi and C++ Builder debuggers; and the NT Symbolic Debugger (NTSD). I'm sure there are many more.

Common Debugging Question: How can I protect my Win32 program from being debugged?

One of the most common requests I get from developers working on vertical market applications with proprietary algorithms is how can they protect those algorithms and keep their competitors from taking a peek at them with a debugger. Although you can call IsDebuggerPresent, which tells you whether a user-mode debugger is running, if someone has half a brain, the first thing she'll do when reverse engineering is patch IsDebuggerPresent to return 0 so that it appears as though no debugger is running.

Although there's no perfect way to protect against a very determined hacker who has physical access to your binaries, you can at least make life a little more different for him at run time. Interestingly enough, the check that IsDebuggerPresent does to see whether a debugger is running on the process has been the same check in all Microsoft operating systems up to the time of this writing. There's no guarantee that it won't change, but the odds are good that it will stay the same for the future.

The next bit of code is a function you can add to your code that does the same thing as IsDebuggerPresent. Of course, just adding that function won't make it impossible to debug your application. To make debugging tougher, you might want to look at interspersing innocuous instructions between the main instructions so that hackers can't simply search for the byte pattern of the IsDebuggerPresent code. A whole book can be written about anti-hacking techniques. However, if you can pass the "two-hour test," which is that it should take longer than two hours for an average developer to try to hack your application, your application is probably safe from all but the most determined and talented hackers.

BOOL AntiHackIsDebuggerPresent ( void ) {     BOOL bRet = TRUE ;     __asm     {         // Get the Thread Information block (TIB).         MOV       EAX , FS:[00000018H]         // 0x30 bytes into the TIB is a pointer field that          // points to a structure related to debugging.         MOV       EAX , DWORD PTR [EAX+030H]         // The second WORD in that debugging structure indicates         // the process is being debugged.         MOVZX     EAX , BYTE PTR [EAX+002H]         // Return the result.         MOV       bRet , EAX      }     return ( bRet ) ; }

Kernel-Mode Debuggers

Kernel-mode debuggers sit between the CPU and the operating system. That means that when you stop in a kernel-mode debugger, the operating system also stops completely. As you can imagine, bringing the operating system to an abrupt halt is helpful when you're working on timing and synchronization problems.

There are three kernel-mode debuggers: the kernel debugger (KD), WinDBG, and SoftICE. I'll briefly describe each of these debuggers in the following sections.

Kernel Debugger (KD)

Windows 2000, Windows XP, and Windows Server 2003 are interesting in that the actual kernel-mode debugger portion is part of NTOSKRNL.EXE, the main kernel file of the operating system. This debugger is available in both the free (release) and checked (debug) builds of the operating system. To turn on kernel-mode debugging for x86-based systems, set the /DEBUG boot option in BOOT.INI and, additionally, the /DEBUGPORT boot option when you need to set the communications port for the kernel-mode debugger to a port other than the default (COM1). KD runs on its own machine, called the host, and communicates with the target machine either through a null modem cable or, optionally, a 1394 (FireWire) cable on Windows XP or Windows Server 2003.

The NTOSKRNL.EXE kernel-mode debugger does just enough to control the CPU so that the operating system can be debugged. The bulk of the debugging work—handling symbols, advanced breakpoints, and disassembly—happens on the KD side. At one time, the Microsoft Windows NT 4 Device Driver Kit (DDK) documented the protocol used across the null modem cable. However, Microsoft no longer documents this protocol.

KD is included in the Debugging Tools for Windows, which is downloadable from http://www.microsoft.com/ddk/debugging. (The current version at the time I wrote this book is also available with this book's sample files.) The power of KD is apparent when you see all the commands it offers for accessing the internal operating system state. If you've ever wanted to see what happens in the operating system, these commands will show you. Having a working knowledge of how Windows device drivers operate will help you follow much of the command's output. Interestingly enough, for all its power, KD is almost never used outside of Microsoft because it's a console application, which makes it tedious to use with source-level debugging. However, for the operating system teams at Microsoft, it's the kernel debugger of choice.

WinDBG

WinDBG is included in the Debugging Tools for Windows. It's a hybrid debugger in that it can be a kernel-mode debugger as well as a user-mode debugger and, with a bit of work, WinDBG lets you debug both kernel-mode and user-mode programs at the same time. For kernel-mode debugging, WinDBG offers all the same power of KD because it shares the same debugging engine as KD. However, WinDBG offers a GUI front end that isn't nearly as easy to use as the Visual Studio .NET debugger, although it is easier to use than KD. With WinDBG, you can debug your device drivers nearly as easily as you would your user-mode applications.

As a user-mode debugger, WinDBG is good, and I strongly recommend that you install it if you haven't already. WinDBG offers more power than the Visual Studio .NET debugger in that WinDBG shows you much more information about your process. However, that power does come at a cost: WinDBG is harder to use than the Visual Studio .NET debugger. Still, I'd advise you to spend some time and effort learning about WinDBG, and I'll show you key highlights and tricks in Chapter 8. The investment might pay off by helping you solve a bug much faster than you could with the Visual Studio .NET debugger alone. On average, I find that I spend about 95 percent of my debugging time in the Visual Studio .NET debugger and the rest in WinDBG.

SoftICE

SoftICE is a commercial kernel-mode debugger from Compuware NuMega and is the only commercial kernel-mode debugger (that I know of) on the market. It's also the only kernel-mode debugger that can operate on a single machine. Unlike the other kernel-mode debuggers, however, SoftICE does an excellent job debugging user-mode programs. As I mentioned earlier, kernel-mode debuggers sit between the CPU and the operating system; SoftICE also sits between the CPU and the operating system when debugging a user-mode program, thereby stopping the entire operating system dead.

At first glance, you might not be impressed by the fact that SoftICE can bring the operating system to a halt. But consider this question: What happens if you have some timing-dependent code you need to debug? If you're using an API function such as SendMessageTimeout, you can easily time out as you step through another thread with a typical GUI debugger. With SoftICE, you can step all you want because the timer that SendMessageTimeout relies on won't be executing while you're running under SoftICE. SoftICE is the only debugger that allows you to effectively debug multithreaded applications. The fact that SoftICE stops the entire operating system when it's active means that solving timing problems is far easier.

Another benefit of SoftICE sitting between the CPU and the operating system is that debugging cross-process interactions becomes very easy. If you're doing COM programming with multiple out-of-process servers, you can easily set breakpoints in all the processes and step between them. Finally, if you do need to step from user mode to kernel mode and back, SoftICE makes such shifting trivial.

The other major advantage that SoftICE has over all other debuggers is that it has a phenomenal collection of informational commands that let you see virtually everything that's happening in the operating system. Although KD and WinDBG have a substantial number of these commands, SoftICE has many more. You can view almost anything in SoftICE, from the state of all synchronization events, to complete HWND information, to extended information about any thread in the system. SoftICE can tell you anything that's happening on your system.

As you might expect, all this wonderful raw power has a price tag. SoftICE, like any kernel-mode debugger, has a steep learning curve because it's essentially its own operating system when it's running. However, your return on investment makes learning how to use SoftICE worth the effort.