Chapter 4: Operating System Debugging Support and How Win32 Debuggers Work

Overview

Learning how your tools operate is a crucial part of this business. If you understand the capabilities and limitations of your tools, you can maximize their return and thus spend less time debugging. Most of the time, debuggers help you out tremendously, but sometimes they can cause subtle problems that baffle you. What makes native code debugging even more interesting is that the operating systems jump in and change the behavior of your processes because they are running under a debugger. Additionally, there's some extremely interesting support inside the operating system to assist you in certain difficult debugging situations. In this chapter, I'll explain what a debugger is, demonstrate how various debuggers operate in Microsoft Win32 operating systems, and discuss the tricks necessary to master the Win32 debugging environment.

After providing a brief overview of the Win32 debuggers at your disposal, I'll go over in detail the special features available when your process is running under a debugger. To show you how debuggers actually work, I'll present two debuggers, the source code of which is with this book's sample files. The first debugger, MinDBG, does just enough to call itself a debugger. The second, WDBG, is a real Win32 debugger sample that does everything a real debugger is supposed to, including manipulating symbol tables to show locals and structures, handling breakpoints, generating disassembly, and coordinating with a graphical user interface (GUI). In discussing WDBG, I'll also cover topics such as how breakpoints work and what all the different symbol files types are and what they mean. Finally, I'll talk about the extra cool symbol engine wrapper I wrote, which makes manipulating locals and arguments simple. The symbol engine was some of the hardest code I wrote for this book, and I'm sure you'll find it very useful!

Why Isn't There a Chapter on .NET Debuggers?

You might be wondering why there isn't a corresponding chapter in this book on how Microsoft .NET debuggers work. Originally, I had intended to write that chapter, but as I was researching the .NET Debugging API, I realized that unlike Win32 debuggers, which are nearly undocumented, the .NET run-time team did a tremendous job documenting the .NET debugging interface. Additionally, the debugger sample provided shows how to do everything a .NET debugger is supposed to do. The sample is about 98 percent of the source code console debugger, CORDBG. (The only part that's missing is the native code disassembler commands.) I spent a couple of weeks working on a .NET debugger, and I quickly realized that I was going to be doing nothing more than rehashing the excellent .NET documentation and that I wouldn't be showing anything that wasn't already shown in the CORDBG sample. The Microsoft Word files, Debug.doc and DebugRef.doc, that describe the .NET Debugging API are already installed on your computer as part of the Visual Studio .NET installation and are in the <Visual Studio .NET Installation Directory>\SDK\v1.1\Tool Developers Guide\Docs directory.

Finally, before jumping into the chapter, I want to make sure I define two standard terms that I'll be using throughout this book: debugger and debuggee. Simply put, a debugger is a process that can control another process in a debugging relationship, and a debuggee is a process started under a debugger. Some operating systems refer to the debugger as the parent process and the debuggee as the child process.