Windows 2000 Operating System Support for Debuggees

[Previous] [Next]

In addition to defining the API that a debugger must call in order to be a debugger, Windows 2000 provides a few other features that help you find problems with your applications. Some of these features aren't that well known and can be confusing the first time you encounter them.

Windows 2000 Heap Checking

As an application starts under a debugger, Windows 2000 turns on the operating system debug heap checking. This heap isn't the C run-time library debug heap; it's the Windows 2000 heap code for those heaps created with the HeapCreate API function. The C run-time library heap is a separate entity, and we'll examine it in Chapter 15. Because processes use the Windows 2000 heaps extensively, you might see information related to heaps in your normal operation, which is why it's important to cover them here. If you attach a debugger to your application rather than start your application under the debugger, you won't activate the Windows 2000 debug heap checking.

With the Windows 2000 debug heap checking turned on, your application will run slightly slower because the Windows 2000 debug heap checking will validate the heap when your application calls HeapFree. Listing 4-1 is a sample program that corrupts memory. When you run the code in Listing 4-1 under a debugger, you'll see that it calls DebugBreak twice on the first HeapFree. Additionally, you'll see output such as the following that shows what was wrong. If you run the program outside the debugger, it runs to completion without reporting any problems.

 HEAP[Heaper.exe]: Heap block at 00441E98 modified at 00441EAA past     requested size of a HEAP[Heaper.exe]: Invalid Address specified to     RtlFreeHeap( 440000, 441ea0)

If you're using your own Windows 2000 heaps, you can turn on some additional flags to receive more diagnostic output. The Platform SDK includes a small utility named GFLAGS.EXE. With it, you can set some of the global flags that Windows 2000 checks when it first starts an application. Figure 4-1 shows GFLAGS.EXE set up for HEAPER.EXE, which is the program in Listing 4-1. Many of the System Registry and Kernel Mode options are global, so you need to be careful if you set them because they can have a major impact on system performance. Setting Image File Options, as shown in Figure 4-1, is much safer because the settings are limited to just that one module.

Listing 4-1 Windows 2000 heap corruption example

 void main(void) {     // Create an operating system heap.     HANDLE hHeap = HeapCreate ( 0 , 128 , 0 ) ;     // Allocate a 10-byte block.     LPVOID pMem = HeapAlloc ( hHeap , 0 , 10 ) ;     // Write 12 bytes to a 10-byte block (an overwrite).     memset ( pMem , 0xAC , 12 ) ;     // Allocate a new 20-byte block.     LPVOID pMem2 = HeapAlloc ( hHeap , 0 , 20 ) ;     // Underwrite 1 byte on the second block.     char * pUnder = (char *)( (DWORD)pMem2 - 1 );     *pUnder = 'P' ;     // Free the first block. This call to HeapFree will trigger a     // breakpoint from the operating system debug heap code.     HeapFree ( hHeap , 0 , pMem ) ;     // Free the second block. Notice that this call won't report     // a problem.     HeapFree ( hHeap , 0 , pMem2 ) ;     // Free a bogus block. Notice that this call won't report a problem.     HeapFree ( hHeap , 0 , (LPVOID)0x1 ) ;     HeapDestroy ( hHeap ) ; }

If you set the same flags that I checked in Figure 4-1, you'll see much more output when you run the same program. Fortunately, the explanations of the flags I've checked are obvious. If you do turn on the flags and run the program, you'll get the same breakpoints, but the output, as shown here, is a little more verbose.

Figure 4-1 GFLAGS.EXE

 PAGEHEAP: process 0x490 created debug heap 00430000     (flags 0x1, 50, 25, 0, 0) PAGEHEAP: process 0x490 created debug heap 00CF0000     (flags 0x1, 50, 25, 0, 0) PAGEHEAP: process 0x490 created debug heap 01600000     (flags 0x1, 50, 25, 0, 0) PAGEHEAP: Tail fill corruption detected:   Allocation at 0x01606FF0   Requested size 0x0000000A   Allocated size 0x00000010   Corruption at 0x01606FFA PAGEHEAP: Attempt to reference block which is not allocated 

Because I'm on the subject of GFLAGS.EXE, I want to take a moment to point out one very useful option, Show Loader Snaps. If you check that box and run your application, you'll see where Windows 2000 is loading the DLLs and how it's going about fixing up the imports, called snapping, for your application. If you need to see exactly what the Windows 2000 loader is doing when it loads your application (in case you have a problem), you'll find it helpful to turn this option on. For more information on loader snaps, see Matt Pietrek's "Under the Hood" column in the September 1999 Microsoft Systems Journal.

Automatically Starting in a Debugger

Some of the hardest types of applications to debug are those started by another process. Windows 2000 services and COM out-of-process servers fall into this category. In many cases, you can use the DebugBreak API function to force a debugger to attach to your process. In two instances, however, using DebugBreak won't work. First, in some cases, DebugBreak won't work with Windows 2000 services. If you need to debug the service startup, calling DebugBreak will get the debugger to attach, but by the time the debugger gets started, the service timeout limit might be reached and Windows 2000 will stop your service. Second, DebugBreak won't work when you need to debug a COM out-of-process server. If you call DebugBreak, the COM error handling will catch the breakpoint exception and terminate your COM out-of-process server. Fortunately, Windows 2000 lets you specify that your application should start in a debugger. This feature allows you to start debugging right from the first instruction. Before you enable this feature for a Windows 2000 service, however, make sure to configure your service to allow interaction with the desktop.

You can enable the start with debugger feature in two ways. The easiest way is to use GFLAGS.EXE and select the Image File Options radio button after typing just your binary's name in the Image File Name edit box. Check the Debugger check box, and type the complete path to the debugger in the edit box next to the Debugger check box.

The other, more difficult, way is to manually set the option with the registry editor. In the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\Current Version\Image File Execution Options key, create a key that is the same as your application's filename. For example, if your application is FOO.EXE, your registry key is FOO.EXE. In your application's registry key, create a new string value named Debugger. In the debugger value, type the complete path and filename to your debugger of choice. If you were using GFLAGS.EXE and you set some global options, you might notice a GlobalFlag string value in your application's key.

Now when you start your application, the debugger automatically starts with your application loaded. If you want to specify any command-line options to the debugger, you can specify them as well in the Debugger value. For example, if you want to use WinDBG and automatically initiate debugging as soon as WinDBG starts, you can fill your Debugger value with "d:\platform sdk\bin\windbg.exe -g."

Quick Break Keys

At times, you need a way to get into the debugger as fast as possible. If you're debugging a console-based application, pressing Ctrl+C or Ctrl+Break will cause a special exception, DBG_CONTROL_C. The DBG_CONTROL_C exception will pop you right into the debugger and allow you to start debugging.

A nice feature of both Windows 2000 and Windows NT 4 is that you can also pop into the debugger at any time for your GUI-based applications. When running under a debugger, by default, pressing the F12 key forces a call to DebugBreak. An interesting aspect of the F12 key processing is that even if you're using F12 as an accelerator or otherwise processing the keyboard messages for F12, you'll still break into the debugger.

The quick break key defaults to F12 in Windows NT 4, but in Windows 2000, you can specify which key to use. In the HKEY_LOCAL_MACHINE\SOFTWARE\ Microsoft\Windows NT\CurrentVersion\AeDebug registry key, you can set the UserDebuggerHotKey value to the VK_* value you want to use. For example, if you want to use the Scroll Lock key to break into the debugger, you'd set the UserDebuggerHotKey value to 0x91. After you set a new value, you must reboot the computer for the change to take effect.