Windows Operating System Support for Debuggees

In addition to defining the API that a debugger must call in order to be a debugger, Windows provides a few other features that help you find problems with your applications. Some of these features aren't that well known and can be confusing the first time you encounter them.

Just-In-Time (JIT) Debugging

Although some of the Visual Studio .NET marketing materials make it sound like Visual Studio is performing the magic behind JIT debugging, the magic is really occurring in the operating system. When an application crashes, Windows looks in the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug to determine what debugger it should call to debug the application. If no values are in the key, Windows XP reports the standard crash dialog box, and Windows 2000 shows a message box and the address of the crash. If the key is present and the appropriate values are filled, under Windows XP, the Debug button in the lower left corner is active so that you can debug the application. Under Windows 2000, the Cancel button is available so that you can start debugging the crash.

The three important values used by JIT debugging in the AeDebug key are the following:

• Auto

• UserDebuggerHotKey

• Debugger

If Auto is set to 0 (zero), the operating system generates the standard crash dialog box and enables the Cancel button so that you can attach the debugger. If Auto is set to 1 (one), the debugger is automatically started. If you want to drive your coworkers crazy, sneak onto their systems and set Auto to 1—they'll have no idea why the debugger is starting each time an application crashes. The UserDebuggerHotKey value identifies the key that is used to break into the debugger. (I'll discuss its usage momentarily.) Finally, the most important key, the Debugger value, specifies the debugger that the operating system will start on the crashed application. The only requirement for the debugger is that it supports attaching to a process. After I discuss the UserDebuggerHotKey value, I'll explain more about the Debugger value and its format.

Quick Break Keys with the UserDebuggerHotKey

At times, you need a way to get into the debugger as fast as possible. If you're debugging a console-based application, pressing Ctrl+C or Ctrl+Break will cause a special exception, DBG_CONTROL_C. The DBG_CONTROL_C exception will pop you right into the debugger and allow you to start debugging.

A nice feature of Windows operating systems is that you can also pop into the debugger at any time for your GUI-based applications. When running under a debugger, by default, pressing the F12 key forces a call to DebugBreak nearly the instant the key is pressed. An interesting aspect of the F12 key processing is that even if you're using F12 as an accelerator or otherwise processing the keyboard messages for F12, you'll still break into the debugger.

The quick break key defaults to F12 but, if you like, you can specify which key to use. The UserDebuggerHotKey value is the numeric VK_* value you use to make the key the debugger hot key. For example, if you want to use the Scroll Lock key to break into the debugger, set the UserDebuggerHotKey value to 0x91. After you set a new value, you must reboot the computer for the change to take effect. A wonderful joke to play on your coworkers is to change the UserDebuggerHotKey to 0x45 (the letter E) so that every time they press the E key, the key breaks into the debugger. However, I take no responsibility if your coworkers gang up on you and make your life miserable.

The Debugger Value

Under the AeDebug key is the Debugger value, where the main action takes place. If you look at the value that's entered for Debugger for a freshly installed operating system, it looks like a string passed to the wsprintf API function: drwtsn32 -p %ld -e %ld -g. That's exactly what it is. The -p is the process ID for the crashing process, and the -e is an event handle value that the debugger needs to signal when its debug loop gets the first thread exit debug event. Signaling the event handle tells the operating system that the debugger attached cleanly. The –g tells Dr. Watson to continue executing the program after attaching.

At any time, you can change the Debugger value to invoke a different debugger. To set Visual Studio .NET as the native debugger, open Visual Studio .NET and select Options from the Tools menu. In the Options dialog box, Debugging folder, select the Just-In-Time property page and ensure that a check mark is next to the Native entry in the list box. You can set WinDBG or Dr. Watson as your debugger of choice by executing WinDBG –I (note that the switch is case sensitive) or DRWTSN32 –I from any command prompt. After you change the Debugger value, make sure to shut down Task Manager if it's running. Task Manager caches the AeDebug key when it starts, so if you attempt to debug a process from the Task Manager Processes page, the debugging might not work if the prior debugger was Visual Studio .NET.

Choosing the Debugger at Crash Time

It's great to have JIT debugging bring up the debugger when your application crashes, but the big limitation is that you can have only a single debugger at a time in the Debugger value. As we'll see in later chapters, the various debuggers you have at your disposal have strengths and weaknesses depending on particular debugging situations. Nothing is worse than having the wrong debugger pop up when you really want the one you know will simplify solving that bug you've spent weeks trying to duplicate.

This was a problem begging to be solved, so I set out to do so. However, because of what look like bugs in the way JIT debugging starts under Visual Studio .NET, I had to work through a lot of trial and error to get my approach working. Before I can discuss the problems, I need to show you how Debugger Chooser, or DBGCHOOSER for short, works.

The idea behind DBGCHOOSER is that it acts as a shim program that gets called when the debuggee crashes and passes on to the real debugger the information necessary to debug the application. To set up DBGCHOOSER, first copy it to a directory on your computer where it won't be accidentally deleted. The operating system tries to start the debugger executable specified in the Debugger value of the AeDebug key, so if the debugger isn't available, you won't get any chance to debug the crash. To initialize DBGCHOOSER, simply run it. Figure 4-1 shows the configuration dialog box in which you set the appropriate commands for each of the various debuggers supported by DBGCHOOSER. The initial run of DBGCHOOSER is set with the appropriate defaults for most developer machines. If you don't have the various debuggers in the path, you can specify the appropriate paths. Pay special attention to the Visual Studio .NET debugger path because the JIT shell used by Visual Studio .NET isn't in the path by default. After you click OK in the configuration dialog box, DBGCHOOSER writes your debugger setting to an INI file stored in your Windows directory and sets itself as the default debugger in the AeDebug key of the registry.

Figure 4-1: The DBGCHOOSER configuration dialog box

After you experience what I hope is one of your rare crashes, you'll see the actual chooser dialog box pop up when you click Debug in the crash dialog boxes, as shown in Figure 4-2. Simply select the debugger you want to use and start debugging.

Figure 4-2: The DBGCHOOSER debugger chooser dialog box

The implementation of DBGCHOOSER is nothing exciting. The first point of interest is that when I called CreateProcess on the debugger the user chose, I had to ensure that I set the inherit handles flag to TRUE. To ensure everything is copacetic with the handles, I have DBGCHOOSER wait on the spawned debugger until it ends. That way I know any necessary inherited handles are still there for the debugger. Although it was more of a pain to figure out than interesting to implement, getting Visual Studio .NET to work properly from DBGCHOOSER took some doing. Everything worked like a champ with WinDBG, Microsoft Visual C++ 6, and Dr. Watson. However, when I'd start Visual Studio .NET (actually VS7JIT.EXE, which in turn spawns the Visual Studio .NET debugger), a message box would pop up indicating that JIT debugging was disabled and wouldn't start debugging.

At first I was a little stumped about what was going on, but a quick check with the wonderful registry monitor program (Regmon), from Mark Russinovich and Bryce Cogswell at www.sysinternals.com, showed me that VS7JIT.EXE was checking the AeDebug key's Debugger value to see whether it was set as the JIT debugger. If it wasn't, up popped the JIT debugging disabled message box. I was able to verify that this was the case by stopping DBGCHOOSER in the debugger while DBGCHOOSER was active because of a crash, and changing the Debugger key back so that it pointed to VS7JIT.EXE. I have no idea why VS7JIT.EXE feels it's so important that it won't debug unless the debugger is the JIT debugger. I did a little quick coding in DBGCHOOSER to fake out VS7JIT.EXE by changing the Debugger value to VS7JIT.EXE before I spawned it, and all was good with the world. To get DBGCHOOSER.EXE reset as the JIT debugger, I spawned a thread that waits for 5 seconds and resets the Debugger value.

As I mentioned when I first started talking about DBGCHOOSER, my solution isn't perfect because of problems with JIT debugging in Visual Studio .NET. On Windows XP, I was testing all permutations of starting and running Visual Studio .NET and found that VS7JIT.EXE stopped running. After playing with it a bit, I found that two instances of VS7JIT.EXE actually run when starting Visual Studio .NET as the JIT debugger. One instance spawns the Visual Studio .NET IDE, and the other instance runs under the RPCSS DCOM server. On rare occasions, only during testing with the supplied implementation, I got the system into states where attempting to spawn VS7JIT.EXE would fail because the DCOM instance wouldn't start. I mainly ran into this problem when I worked on the code to reset the Debugger value in the AeDebug key. Once I settled on the current way of implementing DBGCHOOSER, I've run into the problem just once or twice and only when I set up test cases in which multiple process crashes were happening all at once. I wasn't able to track down the exact cause, but I've never seen this problem in normal work.

Automatically Starting in a Debugger (Image File Execution Options)

Some of the hardest types of applications to debug are those started by another process. Windows services and COM out-of-process servers fall into this category. In many cases, you can use the DebugBreak API function to force a debugger to attach to your process. In two instances, however, using DebugBreak won't work. First, in some cases, DebugBreak won't work with Windows services. If you need to debug the service startup, calling DebugBreak will get the debugger to attach, but by the time the debugger gets started, the service timeout limit might be reached and Windows will stop your service. Second, DebugBreak won't work when you need to debug a COM out-of-process server. If you call DebugBreak, the COM error handling will catch the breakpoint exception and terminate your COM out-of-process server. Fortunately, Windows lets you specify that your application should start in a debugger. This feature allows you to start debugging right from the first instruction. Before you enable this feature for a Windows service, however, make sure to configure your service to allow interaction with the desktop.

The best way to set up automatic debugging is to manually set the option with the registry editor. In the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options key, create a key that is the same as your application's filename. For example, if your application is FOO.EXE, your registry key is FOO.EXE. In your application's registry key, create a new string value named Debugger. In the Debugger value, type the complete path and filename to your debugger of choice.

Now when you start your application, the debugger automatically starts with your application loaded. If you want to specify any command-line options to the debugger, you can specify them as well in the Debugger value. For example, if you want to use WinDBG and automatically initiate debugging as soon as WinDBG starts, you can fill your Debugger value with "d:\windbg\windbg.exe -g".

To use Visual Studio .NET as your debugger of choice, you'll have to do a bit more work. The first problem is that Visual Studio .NET cannot debug an executable without a solution file. If you're developing the executable (in other words, you have a solution with source code), you can use that solution. However, the last build open will be the build run. Therefore, if you want to debug the release build or a binary you don't have source code to, open the project, set the active solution configuration to Release, and close the solution. If you don't have a solution file for the executable, click the File, Open Solution menu option and open the executable as the solution. Start debugging the solution and save the solution file when prompted.

Once you have the solution you'll use worked out, the command line you'll set in the Debugger value would look like the following. Unless you've manually added the Visual Studio .NET directory, <VS.NET Installation Dir>\ Common7\IDE, to the system PATH environment variable, you'll need to specify the complete drive and directory with DEVENV.EXE. The /run command-line option to DEVENV.EXE tells it to start debugging the solution passed on the command line.

g:\vsnet\common7\ide\devenv /run d:\disk\output\wdbg.sln

The second problem you'll get to deal with is that the Debugger string value can be only 65 characters long. If you've installed Visual Studio .NET into the default paths, the path will almost certainly be too long to use. What you'll need to do is get creative with the SUBST command and assign the paths to DEVENV.EXE and your solution to drive letters.

Some of you old-timers out there might remember that you can set the Debugger key from GFLAGS.EXE, a small utility that comes with WinDBG. Unfortunately, GFLAGS.EXE is broken and will accept only a command-line string of 25 characters for the Debugger value. Consequently, it's still best to create the process key and Debugger value manually.

Common Debugging Question: My boss is sending me so much e-mail that I can't get anything done. Is there any way I can slow down the dreaded PHB e-mail?

Although many of your bosses mean well, their incessant e-mail messages can become distracting and keep you from the real work you need to do. Fortunately, there's a simple solution that works quite well and will give you a week or so of wonderful peace so that you can work at hitting your deadlines. The less technically proficient your boss and network administrators are, the more time you'll get.

In the previous section, I talked about the Image File Execution Options section of the registry and the fact that whenever you set a process's Debugger value, that process automatically starts under that debugger. The trick to ending the PHB (pointy haired boss) mail is the following:

1. Walk into your boss's office.

2. Open REGEDIT.EXE. If your boss is currently in the office, explain that you need to run a utility on his machine so that he can access the XML Web services you're building. (It doesn't really matter whether you're creating XML Web services—the buzzwords alone will cause your boss to readily let you mess with his machine.)

3. In the Image File Execution Options section, create a key OUTLOOK.EXE. (Substitute the executable name of your e-mail program if you don't use Microsoft Outlook.) Tell your boss that you're doing this to allow him to have e-mail access to XML Web services.

4. Create the Debugger value and set the string to SOL.EXE. Indicate to your boss that SOL is to allow your XML Web services to access Sun Solaris machines, so it's necessary you use it.

5. Close REGEDIT.EXE.

6. Tell your boss that he's all set and can now start accessing XML Web services. The real trick at this point is to keep a straight face while walking out of your boss's office.

Avoiding laughter during this experiment is much more difficult than it sounds, so you might want to practice these steps with a few coworkers first.

What you've just set up is a situation in which every time your boss starts Outlook, Solitaire runs instead. (Since most bosses spend their days playing Solitaire anyway, your boss will be sidetracked for a couple of games before he realizes that he meant to start Outlook.) Eventually, he'll continue to click the Outlook icon until so many copies of Solitaire are running that he'll run out of virtual memory and have to reboot his machine. After a couple of days of this click-a-million-times-and-reboot cycle, your boss will eventually have a network administrator come in and look at his machine.

The admin will get all excited because she has a problem that is a little more interesting than helping the folks in accounts receivable reset their passwords. The admin will play with the machine for at least a day in your boss's office, thus keeping your boss from even being close to a machine. If anyone asks your opinion, the stock answer is, "I've heard of strange interaction problems between EJB and NTFS across the DCOM substrata architecture necessary to access the MFT using the binary least squares sort algorithm." The admin will take your boss's machine back to her office and continue to play with it for a couple of days. Eventually, the admin will repave the hard disk and reinstall everything on the machine, taking another day or so. By the time your boss gets his machine back, he'll have four days of e-mail to get through, so it will be at least a day before he gets out from under all that mail, and you can safely ignore those messages for another day or two. If the PHB-mail starts getting thick again, simply repeat the steps.

Important note: Use this technique at your own risk.