WDBG: A Real Debugger

[Previous] [Next]

I thought the best way to show you how a debugger worked was to write one, so I did. Although WDBG might not replace the Visual C++ debugger any time soon, it certainly does most of the things a debugger is supposed to do. If you look at Figure 4-2, you'll see WDBG debugging Microsoft Word. In the figure, Word is stopped at a breakpoint I set on GetProcAddress. The Memory window, the one in the upper right-hand corner, is showing the second parameter that Word passed to this particular instance of GetProcAddress, the string PHevCreateFileInfo. As you look around Figure 4-2, you'll see that WDBG takes care of the business you'd expect a debugger to tend to, including showing registers, viewing call stacks, disassembling code, and showing the currently loaded modules and the currently running threads. What you don't see in the picture but what will become apparent when you first run WDBG is that WDBG also supports breakpoints, symbol enumeration, and breaking the application to stop in the debugger.

Figure 4-2 WDBG in action

Overall, I'm happy with WDBG because it's a good sample. Looking at the WDBG user interface (UI), however, you can see that I didn't spend a great deal of time fiddling with the UI portions. In fact, all the multiple-document interface (MDI) windows in WDBG are edit controls. That was intentional—I kept the UI simple because I didn't want UI details to distract you from the essential debugger code. I wrote the WDBG UI using the Microsoft Foundation Class (MFC) library, so if you're so inclined, you shouldn't have any trouble designing a spiffier UI.

Before moving into the specifics of debugging, let's take a closer look at WDBG. Table 4-2 lists all the main subsystems of WDBG and describes what they do. One of my intentions in creating WDBG was to define a neutral interface between the UI and the debug loop. With a neutral interface, if I wanted to make WDBG.EXE support remote debugging over a network, I'd just have to replace the local debugging DLLs.

Table 4-2 WDBG Main Subsystems

Subsystem	Description
WDBG.EXE	This module contains all the UI code. Additionally, all the breakpoint processing is taken care of here. Most of this debugger's work occurs in WDBGPROJDOC.CPP.
LOCALDEBUG.DLL	This module contains the debug loop. Because I wanted to be able to reuse this debug loop, the user code, WDBG.EXE in this case, passes a C++ class derived from CDebugBaseUser (defined in DEBUGINTERFACE.H) to the debug loop. The debug loop will call into that class when any of the debugging events occurs. The user's class is responsible for all synchronization. For WDBG.EXE, WDBGUSER.H and WDBGUSER.CPP contain the coordinating class. WDBG.EXE uses simple SendMessage synchronization. In other words, the debug thread sends a message to the UI thread and blocks until the UI thread returns. If the debugging event is one that required user input, the debug thread blocks after the send message on a synchronization event. Once the UI thread processes the Go command, it sets the synchronization event and the debug thread starts running again.
LOCALASSIST.DLL	This simple module is just a wrapper around the API functions for manipulating the debuggee's memory and registers. By using the interface defined in this module, WDBG.EXE and I386CPUHELP.DLL can instantly handle remote debugging just by replacing this module.
I386CPUHELP.DLL	This module is the IA32 (Pentium) helper module. Although this module is specific to Pentium processors, its interface, defined in CPUHELP.H, is CPU-independent. If you wanted to port WDBG to a different processor, this module is the only one you should have to replace. The disassembler in this module came from the Dr. Watson sample code that ships on the Platform SDK. Although the disassembler works, it appears to need updating to support the later Pentium CPU variants.

Reading and Writing Memory

Reading from a debuggee's memory is simple. ReadProcessMemory takes care of it for you. A debugger has full access to the debuggee if the debugger started it because the handle to the process returned by the CREATE_PROCESS_DEBUG_EVENT debug event has PROCESS_VM_READ and PROCESS_VM_WRITE access. If your debugger attaches to the process with DebugActiveProcess, you must use OpenProcess to get a handle to the debuggee, and you need to specify both read and write access.

Before I can talk about writing to the debuggee's memory, I need to briefly explain an important concept: copy-on-write. When Windows loads an executable file, Windows shares as many mapped memory pages of that binary as possible with the different processes using it. If one of those processes is running under a debugger and one of those pages has a breakpoint written to it, the breakpoint obviously can't be present in all the processes sharing that page. As soon as any process running outside the debugger executed that code, it would crash with a breakpoint exception. To avoid that situation, the operating system sees that the page changed for a particular process and makes a copy of that page that is private to the process that had the breakpoint written to it. Thus, as soon as a process writes to a page, the operating system copies the page.

Writing to the debuggee memory is almost as straightforward as reading from it. Because the memory pages you want to write to might be marked as read-only, however, you first need to call VirtualQueryEx to get the current page protections. Once you have the protections, you can use the VirtualProtectEx API function to set the page to PAGE_EXECUTE_READWRITE so that you can write to it and Windows is prepared to do the copy-on-write. After you do the memory write, you'll need to set the page protection back to what it originally was. If you don't, the debuggee might accidentally write to the page and succeed when it should fail. If the original page protections were read-only, the debuggee's accidental write would lead to an access violation. By forgetting to set the page protection back, the accidental write wouldn't generate the exception and you'd have a case in which running under the debugger is different from running outside the debugger.

An interesting detail about the Win32 Debugging API is that the debugger is responsible for getting the string to output when an OUTPUT_DEBUG_STRING_EVENT comes through. The information passed to the debugger includes the location and the length of the string. When it receives this message, the debugger goes and reads the memory out of the debuggee. In Chapter 3, I mentioned that trace statements could easily change your application's behavior when running under a debugger. Because all threads in the application stop when the debug loop is processing an event, calling OutputDebugString in the debuggee means that all your threads stop. Listing 4-3 shows how WDBG handles the OUTPUT_DEBUG_STRING_EVENT. Notice that the DBG_ReadProcessMemory function is the wrapper function around ReadProcessMemory from LOCALASSIST.DLL.

Listing 4-3 OutputDebugStringEvent from PROCESSDEBUGEVENTS.CPP

 static DWORD OutputDebugStringEvent ( CDebugBaseUser * pUserClass , LPDEBUGGEEINFO pData , DWORD dwProcessId , DWORD dwThreadId , OUTPUT_DEBUG_STRING_INFO & stODSI ) { TCHAR szBuff[ 512 ] ; HANDLE hProc = pData->GetProcessHandle ( ) ; DWORD dwRead ; // Read the memory. BOOL bRet = DBG_ReadProcessMemory( hProc , stODSI.lpDebugStringData , szBuff , min ( sizeof ( szBuff ) , stODSI.nDebugStringLength ), &dwRead ); ASSERT ( TRUE == bRet ) ; if ( TRUE == bRet ) { // Always NULL terminate the string. szBuff [ dwRead + 1 ] = _T ( '\0' ) ; // Convert CR/LFs if I m supposed to. pUserClass->ConvertCRLF ( szBuff , sizeof ( szBuff ) ) ; // Send the converted string on to the user class. pUserClass->OutputDebugStringEvent ( dwProcessId , dwThreadId , szBuff ) ; } return ( DBG_CONTINUE ) ; } 

Breakpoints and Single Stepping

Most engineers don't realize that debuggers use breakpoints extensively behind the scenes to allow the debugger to control the debuggee. Although you might not directly set any breakpoints, the debugger will set many to allow you to handle tasks such as stepping over a function call. The debugger also uses breakpoints when you choose to run to a specific source file line and stop. Finally, the debugger uses breakpoints to break into the debuggee on command (via the Debug Break menu option in WDBG, for example).

The concept of setting a breakpoint is simple. All you need to do is have a memory address where you want to set a breakpoint, save the opcode (the value) at that location, and write the breakpoint instruction into the address. On the Intel Pentium family, the breakpoint instruction mnemonic is "INT 3" or an opcode of 0xCC, so you need to save only a single byte at the address you're setting the breakpoint. Other CPUs, such as the Intel Merced, have different opcode sizes, so you would need to save more data at the address.

Listing 4-4 shows the code for the SetBreakpoint function. As you read through this code, keep in mind that the DBG_* functions are those that come out of LOCALASSIST.DLL and help isolate the various process manipulation routines, making it easier to add remote debugging to WDBG. The SetBreakpoint function illustrates the processing (described earlier in the chapter) necessary for changing memory protection when you're writing to it.

Listing 4-4 SetBreakpoint from 1386CPUHELP.C

 int CPUHELP_DLLINTERFACE __stdcall SetBreakpoint ( PDEBUGPACKET dp , ULONG ulAddr , OPCODE * pOpCode ) { DWORD dwReadWrite = 0 ; BYTE bTempOp = BREAK_OPCODE ; BOOL bReadMem ; BOOL bWriteMem ; BOOL bFlush ; MEMORY_BASIC_INFORMATION mbi ; DWORD dwOldProtect ; ASSERT ( FALSE == IsBadReadPtr ( dp , sizeof ( DEBUGPACKET ) ) ) ; ASSERT ( FALSE == IsBadWritePtr ( pOpCode , sizeof ( OPCODE ) ) ) ; if ( ( TRUE == IsBadReadPtr ( dp , sizeof ( DEBUGPACKET ) ) ) || ( TRUE == IsBadWritePtr ( pOpCode , sizeof ( OPCODE ) ) ) ) { TRACE0 ( "SetBreakpoint : invalid parameters\n!" ) ; return ( FALSE ) ; } // If the operating system is Windows 98 and the address is above // 2 GB, just leave quietly. if ( ( FALSE == IsNT ( ) ) && ( ulAddr >= 0x80000000 ) ) { return ( FALSE ) ; } // Read the opcode at the location. bReadMem = DBG_ReadProcessMemory ( dp->hProcess , (LPCVOID)ulAddr , &bTempOp , sizeof ( BYTE ) , &dwReadWrite ) ; ASSERT ( FALSE != bReadMem ) ; ASSERT ( sizeof ( BYTE ) == dwReadWrite ) ; if ( ( FALSE == bReadMem ) || ( sizeof ( BYTE ) != dwReadWrite ) ) { return ( FALSE ) ; } // Is this new breakpoint about to overwrite an existing // breakpoint opcode? if ( BREAK_OPCODE == bTempOp ) { return ( -1 ) ; } // Get the page attributes for the debuggee. DBG_VirtualQueryEx ( dp->hProcess , (LPCVOID)ulAddr , &mbi , sizeof ( MEMORY_BASIC_INFORMATION ) ) ; // Force the page to copy-on-write in the debuggee. if ( FALSE == DBG_VirtualProtectEx ( dp->hProcess , mbi.BaseAddress , mbi.RegionSize , PAGE_EXECUTE_READWRITE , &mbi.Protect ) ) { ASSERT ( !"VirtualProtectEx failed!!" ) ; return ( FALSE ) ; } // Save the opcode I m about to whack. *pOpCode = (void*)bTempOp ; bTempOp = BREAK_OPCODE ; dwReadWrite = 0 ; // The opcode was saved, so now set the breakpoint. bWriteMem = DBG_WriteProcessMemory ( dp->hProcess , (LPVOID)ulAddr , (LPVOID)&bTempOp , sizeof ( BYTE ) , &dwReadWrite ) ; ASSERT ( FALSE != bWriteMem ) ; ASSERT ( sizeof ( BYTE ) == dwReadWrite ) ; if ( ( FALSE == bWriteMem ) || ( sizeof ( BYTE ) != dwReadWrite ) ) { return ( FALSE ) ; } // Change the protection back to what it was before I blasted the // breakpoint in. VERIFY ( DBG_VirtualProtectEx ( dp->hProcess , mbi.BaseAddress , mbi.RegionSize , mbi.Protect , &dwOldProtect ) ) ; // Flush the instruction cache in case this memory was in the CPU // cache. bFlush = DBG_FlushInstructionCache ( dp->hProcess , (LPCVOID)ulAddr , sizeof ( BYTE ) ) ; ASSERT ( TRUE == bFlush ) ; return ( TRUE ) ; } 

After you set the breakpoint, the CPU will execute it and will tell the debugger that an EXCEPTION_BREAKPOINT (0x80000003) occurred—that's where the fun begins. If it's a regular breakpoint, the debugger will locate and display the breakpoint location to the user. After the user decides to continue execution, the debugger has to do some work to restore the state of the program. Because the breakpoint overwrote a portion of memory, if you, as the debugger writer, were to just let the process continue, you would be executing code out of sequence and the debuggee would probably crash. What you need to do is to move the current instruction pointer back to the breakpoint address and replace the breakpoint with the opcode you saved when you set the breakpoint. After restoring the opcode, you can continue executing.

There's only one small problem: How do you reset the breakpoint so that you can stop at that location again? If the CPU you're working on supports single-step execution, resetting the breakpoint is trivial. In single-step execution, the CPU executes a single instruction and generates another type of exception, EXCEPTION_SINGLE_STEP (0x80000004). Fortunately, all CPUs that Win32 runs on support single-step execution. For the Intel Pentium family, setting single-step execution requires that you set bit 8 on the flags register. The Intel reference manual calls this bit the TF, or Trap Flag. The code in Listing 4-5 shows the SetSingleStep function and the work needed to set the TF. After replacing the breakpoint with the original opcode, the debugger marks its internal state to reflect that it's expecting a single-step exception, sets the CPU into single-step execution, and then continues the process.

Listing 4-5 SetSingleStep function from I386CPUHELP.C

 BOOL CPUHELP_DLLIMNTERFACE __stdcall     SetSingleStep ( PDEBUGPACKET dp ) { BOOL bSetContext ; ASSERT ( FALSE == IsBadReadPtr ( dp , sizeof ( DEBUGPACKET ) ) ) ; if ( TRUE == IsBadReadPtr ( dp , sizeof ( DEBUGPACKET ) ) ) { TRACE0 ( "SetSingleStep : invalid parameters\n!" ) ; return ( FALSE ) ; } // For the i386, just set the TF bit. dp->context.EFlags |= TF_BIT ; bSetContext = DBG_SetThreadContext ( dp->hThread , &dp->context ) ; ASSERT ( FALSE != bSetContext ) ; return ( bSetContext ) ; } 

After the debugger releases the process by calling ContinueDebugEvent, the process immediately generates a single-step exception after the single instruction executes. The debugger checks its internal state to verify that it was expecting a single-step exception. Because the debugger was expecting a single-step exception, it knows that a breakpoint needs to be reset. The single step caused the instruction pointer to move past the original breakpoint location. Therefore, the debugger can set the breakpoint opcode back at the original breakpoint location. The operating system automatically clears the TF each time the EXCEPTION_SINGLE_STEP exception occurs, so there's no need for the debugger to clear it. After setting the breakpoint, the debugger releases the debuggee to continue running.

If you want to see all the breakpoint processing in action, look for the CWDBGProjDoc::HandleBreakpoint method in the WDBGPROJDOC.CPP file on the companion CD. I defined the breakpoints themselves in BREAKPOINT.H and BREAKPOINT.CPP, and those files contain a couple of classes that handle different styles of breakpoints. I set up the WDBG Breakpoints dialog box so that you could set breakpoints as the debuggee is running, just as you do in the Visual C++ debugger. Being able to set breakpoints on the fly means that you need to keep careful track of the debuggee state and the breakpoint states. See the CBreakpointsDlg::OnOK method in BREAKPOINTSDLG.CPP on the companion CD for details on how I handle enabling and disabling breakpoints, depending on what the debuggee state is.

One of the neater features I implemented in WDBG was the Debug Break menu option. This option means that you can break into the debugger at any time while the debuggee is running. Although WDBG uses the breakpoint operations described earlier in the chapter, the breakpoints used to implement the Debug Break option are referred to as one-shot breakpoints because the breakpoints are removed just as soon as they trigger. Getting those one-shot breakpoints set is pretty interesting. The full details are in CWDBGProjDoc::OnDebugBreak in WDBGPROJDOC.CPP, but I'll go into greater detail here because I think you'll find the explanation enlightening. Listing 4-6 shows the CWDBGProjDoc::OnDebugBreak function from WDBGPROJDOC.CPP. (To find out more about one-shot breakpoints, see the section "Step Into, Step Over, and Step Out" later in this chapter.)

Listing 4-6 Debug Break processing in WDBGPROJDOC.CPP

 void CWDBGProjDoc :: OnDebugBreak ( ) { // Just for my own peace of mind. ASSERT ( m_vDbgThreads.size ( ) > 0 ) ; // The idea here is to get all the debuggee s threads suspended and // set a breakpoint at the current instruction pointer for each. // That way, I can guarantee that at least one of the threads // will trip the one-shot breakpoints. // One situation in which setting a breakpoint on each thread // won t work is when an application is hung. Because no // threads are turning over, the breakpoints never get called. // To make the deadlock case work, I d need to use an algorithm such // as the following: // 1. Set the breakpoints as this function does. // 2. Set a state flag indicating that I m waiting on a Debug Break // breakpoint. // 3. Set a background timer to wait for the breakpoint. // 4. If one of the breakpoints goes off, clear the timer. Life is // good. // 5. If the timer goes off, the application is hung. // 6. After the timer, set the instruction pointer of one of the // threads to another address and put a breakpoint at that // address. // 7. Restart the thread. // 8. When this special breakpoint fires, clear the breakpoint and // reset the instruction pointer back to the original location. // To avoid problems, I ll boost the priority of this thread so // that I get through setting these breakpoints as fast as possible // and keep any of the debuggee s threads from being scheduled. HANDLE hThisThread = GetCurrentThread ( ) ; int iOldPriority = GetThreadPriority ( hThisThread ) ; SetThreadPriority ( hThisThread , THREAD_BASE_PRIORITY_LOWRT ) ; HANDLE hProc = GetDebuggeeProcessHandle ( ) ; DBGTHREADVECT::iterator i ; for ( i = m_vDbgThreads.begin ( ) ; i != m_vDbgThreads.end ( ) ; i++ ) { // Suspend this thread. If it has a suspend count already, I // don t really care. That s why I set a breakpoint on each // thread in the debuggee. I ll hit an active one eventually. DBG_SuspendThread ( i->m_hThread ) ; // Now that the thread is suspended, I can get the context. CONTEXT ctx ; ctx.ContextFlags = CONTEXT_FULL ; // If GetThreadContext fails, I have to handle the error message // carefully. Because this thread s priority is set to real-time, // if I use an ASSERT, the computer might hang on the message // box, so in the else statement, I can indicate the error only // with a trace statement. if ( FALSE != DBG_GetThreadContext ( i->m_hThread , &ctx ) ) { // Find the address that the instruction pointer is about to // execute. That address is where I ll set the breakpoint. DWORD dwAddr = ReturnInstructionPointer ( &ctx ) ; COneShotBP cBP ; // Set the breakpoint. cBP.SetBreakpointLocation ( dwAddr ) ; // Arm it. if ( TRUE == cBP.ArmBreakpoint ( hProc ) ) { // Add this breakpoint to the Debug Break list only if // the breakpoint was successfully armed. The debuggee // could easily have multiple threads sitting on the // same instruction, so I want only one breakpoint set // on that address. m_aDebugBreakBPs.Add ( cBP ) ; } } else { TRACE ( "GetThreadContext failed! Last Error = 0x%08X\n" , GetLastError ( ) ) ; #ifdef _DEBUG // Because GetThreadContext failed, I probably need to take a // look at what happened. Therefore, I ll pop into the // the debugger debugging the WDBG debugger. Even though // the WDBG thread is running at a real-time priority level, // calling DebugBreak will immediately pull this thread out // of the operating system scheduler, so the priority drops. DebugBreak ( ) ; #endif } } // All the threads have breakpoints set. Now I ll restart them all // and post a thread message to each one. The reason for posting // the thread message is simple. If the debuggee is chugging away // on messages or other processing, it will break immediately. // However, if it s just idling in a message loop, I need to give it // a tickle to force it into action. Because I have the thread ID, // I ll just send the thread a WM_NULL message. WM_NULL is supposed // to be a benign message, so it shouldn t screw up the debuggee. If // the thread doesn t have a message queue, this function just fails // for that thread with no harm done. for ( i = m_vDbgThreads.begin ( ) ; i != m_vDbgThreads.end ( ) ; i++ ) { // Let this thread resume so that it hits the breakpoint. DBG_ResumeThread ( i->m_hThread ) ; PostThreadMessage ( i->m_dwTID , WM_NULL , 0 , 0 ) ; } // Now drop the priority back down. SetThreadPriority ( hThisThread , iOldPriority ) ; } 

When you want to stop a debuggee that's churning like mad, you need to get a breakpoint jammed into the CPU instruction stream so that you can stop in the debugger. The question is, What do you need to do to get the instruction in there? If a thread is running, the only thing you can do to get it to a known point is to suspend it by using the SuspendThread API function. Once the thread is suspended, you can look at it with the GetThreadContext API function and determine the current instruction pointer. Once you have the instruction pointer, you're back to setting simple breakpoints. After you set the breakpoint, you need to call the ResumeThread API function so that you can let the thread continue execution and have it hit your breakpoint.

Although breaking into the debugger is fairly simple, you still need to think about a couple of issues. The first issue is that your breakpoint might not trigger. If the debuggee is processing a message or doing some other work, it will break. If the debuggee is sitting there waiting for a message to arrive, however, the breakpoint won't trigger until the debuggee receives a message. Although you could require the user to move the mouse over the debuggee to generate a WM_MOUSEMOVE message, the user might not be too happy about this requirement.

To ensure that the debuggee reaches your breakpoint, you need to send a message to the debuggee. If all you have is a thread handle given to you by the Debugging API, how do you turn the handle into the appropriate HWND? Unfortunately, you can't. However, because you have the thread handle, you can always call PostThreadMessage, which will post a message to the thread message queue. Because the HWND message processing layers on top of the thread message queue, calling PostThreadMessage does exactly what you need it to do.

The only question then becomes, What message do I post? You don't want to post a message that could cause the debuggee to do any real processing, thus allowing the debugger to change the behavior of the debuggee. For example, posting a WM_CREATE message probably wouldn't be a good idea. Fortunately, the WM_NULL message is supposed to be a benign message and is what you're supposed to use in hooks if you change a message. It does no harm to post the WM_NULL message with PostThreadMessage even if the thread doesn't have a message queue. And if the thread doesn't have a message queue, such as in a console application, calling PostThreadMessage doesn't do any damage. Because console-based applications will always be processing, even if waiting for a keystroke, setting the breakpoint at the current executing instruction will cause the break.

Another issue involves multithreading. If you're going to suspend only a single thread and the application is multithreaded, how do you know which thread to suspend? If you suspend and set the breakpoint in the wrong thread, say one that is blocked waiting on an event that is signaled only when background printing occurs, your breakpoint might never go off unless the user decides to print something. If you want to break on a multithreaded application, the only safe course is to suspend all the threads and set a breakpoint in each one.

Suspending all the threads and setting a breakpoint in each one works just great on an application that has only two threads. If you want to break on an application that has many threads, however, you could leave yourself open to a problem. As you're walking through and suspending each of the debuggee's threads, you're changing the state of the application such that it's possible for you to cause the application to deadlock. To get all the threads suspended, the breakpoints set, and the threads resumed without causing problems, the debugger needs to boost its own thread priority. By boosting the priority to THREAD_BASE_PRIORITY_LOWRT, the debugger can have its thread stay scheduled so that the debuggee's threads don't execute as the debugger manipulates them.

So far, my algorithm for breaking in a multithreaded application sounds reasonable. However, the debugger still needs to deal with one last issue to make the Debug Break option work completely. If you have all the breakpoints set in all the threads and you resume the threads, you still face one situation in which the break won't happen. By setting the breakpoints, you're relying on at least one of the threads to execute in order to trigger the breakpoint exception. What do you think happens if the process is in a deadlock situation? Nothing happens—no threads execute and your carefully positioned breakpoints never trigger the exception.

I told you the Debug Break business gets interesting. When you're breaking in a deadlock, you need to set up a timer to mark when you added the break. After your period of time elapses (the Visual C++ debugger uses 3 seconds), you need to take some drastic action. When the Debug Break option times out, you'll need to set one of the thread's instruction pointers to another address, set a breakpoint at that new address, and restart the thread. When that special breakpoint fires, you need to set the thread instruction pointer back to its original location. In WDBG, I didn't implement the anti-deadlock processing, but I left the implementation as an exercise for you in the CWDBGProjDoc::OnDebugBreak function in WDBGPROJDOC.CPP on the companion CD. The complete infrastructure is in place to handle the anti-deadlock processing, and it would probably take no more than a couple hours to put in. By the time you had it implemented, you'd have a good idea how WDBG works.

Symbol Tables, Symbol Engines, and Stack Walking

The real black art to writing a debugger involves symbol engines, the code that manipulates symbol tables. Debugging at the straight assembly-language level is interesting for the first couple of minutes you have to do it, but it gets old quickly. Symbol tables, also called debugging symbols, are what turn hexadecimal numbers into source file lines, function names, and variable names. Symbol tables also contain the type information your program uses. This type information allows the debugger to take raw data and display it as the structures and variables you defined in your program. Dealing with modern symbol tables is difficult because the most commonly used format, Program Database (PDB), is undocumented and its owners don't plan to document it anytime soon. Fortunately, you can get at least partial access to the symbol tables.

The Different Symbol Formats

Before diving into a discussion of accessing symbol tables, I need to go over the various symbol formats available. I've found that people are a little confused about what the different formats are and what they offer, so I want to set the record straight.

The first format, SYM, is an older format that used to be common in the MS-DOS and 16-bit Windows days. The only current use of SYM is for the debugging symbols for Windows 98; the SYM format is used here because most of the core kernel is still 16-bit code. WDEB386 is the only debugger actively using them.

Common Object File Format (COFF) was one of the original symbol table formats and was introduced with Windows NT 3.1, the first version of Windows NT. The Windows NT team was experienced with operating system development and wanted to bootstrap Windows NT with some existing tools. The COFF format is part of a larger specification that different UNIX vendors followed to try to make common binary file formats. Although the whole COFF symbol specification is in WINNT.H, the only parts generated by the Microsoft tools are public functions and global variables. Microsoft used to support source and line information but has been gradually moving away from the COFF format in favor of more modern symbol table formats.

The C7, or CodeView, format first appeared as part of Microsoft C/C++ version 7 back in the MS-DOS days. If you're an old timer, you might have heard the name CodeView before—CodeView was the name of the old Microsoft debugger. The C7 format has been updated to support the Win32 operating systems, and you can still generate this format by using the /Z7 command-line switch to CL.EXE or by selecting C7 Compatible from the Debug Info drop-down list on the C/C++ tab of the Project Settings dialog box. On the Link tab of the Project Settings dialog box, uncheck Use Program Database in the Customize category to turn on the /PDB:NONE linker switch. WinDBG and the Visual C++ debugger both support complete source and line debugging with the C7 format. The C7 format is self-contained in the executable module because the linker appends the symbolic information to the binary after it links. Attaching the symbol information to your binary means that your debugging binaries can be quite large; symbol information can easily be larger than your binary file. If you want to see whether a file contains C7 information, open the binary in a hex editor and move to the end of the file. If you see "NB11" followed by 4 bytes, the file has C7 information in it.

If you're interested in symbol tables and would like to write one, the C7 specification is on MSDN. Look for it in the "VC5.0 Symbolic Debug Information" topic. The specification lists only the raw byte structure and type definitions. If you'd like to see the actual type definitions in C, the Dr. Watson source code, included on the MSDN CDs, has some old C7 format header files in its include directory. Although those header files are considerably dated, they can give you an idea what the structures look like.

Although you could use the C7 format for your applications if you wanted to, you probably shouldn't. The main reason for not using C7 is that it automatically turns off incremental linking. With incremental linking turned off, link times increase dramatically. The other reason for avoiding C7 is that it makes binary files incredibly large. Although you could strip out the symbol information with REBASE.EXE, other formats, namely PDB, automatically remove the symbol information for you.

The PDB format is the most common symbol format used today, and both Visual C++ and Visual Basic support it. Unlike the C7 format, PDB symbols are stored in a separate file or files, depending on how the application is linked. By default, Visual C++ 6 links with /PDBTYPE:SEPT, which puts the type information into VC60.PDB and the symbols into <binary name>.PDB. Separating the type information from the debug symbols makes linking faster and requires less disk space. However, the documentation states that if you're building binaries that others could be debugging, you should use /PDBTYPE:CON so that all the type information and debug symbols are consolidated into a single PDB file. Fortunately, Visual Basic automatically uses /PDBTYPE:CON.

To see whether a binary contains PDB symbol information, open it in a hex editor and move to the end of the file. You'll see a marker to the debugging information. If the marker starts with "NB10" and ends with the complete path to the PDB file produced during the link, the binary includes PDB symbols. The debugging format in PDB still resembles the C7 format internally. However, Microsoft optimized the layout for incremental linking and speed. Unfortunately, the low-level interfaces to the PDB file, which are in MSDBI.DLL, are proprietary and Microsoft hasn't released them.

DBG files are unique because, unlike the other symbol formats, the linker doesn't create them. A DBG file is basically just a file that holds other types of debug symbols, such as COFF or C7. DBG files use some of the same structures defined by the Portable Executable (PE) file format—the format used by Win32 executables. REBASE.EXE produces DBG files by stripping the COFF or C7 debugging information out of a module. There's no need to run REBASE.EXE on a module that was built using PDB files because the symbols are already separate from the module. If you're generating C7 symbols and you need to strip them, read the MSDN documentation on REBASE.EXE to see how to do it. Microsoft distributes the operating system debugging symbols in DBG files, and with Windows 2000, the PDB files are included as well. Before you get your hopes up that the operating system symbols include everything you need to reverse engineer the entire operating system, let me warn you that the files include only the public and global information. Using these files makes it much easier to see where you are when you're dropped into the middle of the Disassembly window.

If you're interested in symbol engines and you start researching how to write one, in your studies you'll run across one other symbol type, OMAP. It appears only in some Microsoft applications, and you can sometimes see it when you use the DUMPBIN.EXE utility with the /SYMBOLS option to dump the symbol information. (DUMPBIN.EXE comes with Visual C++.) OMAP is a completely undocumented symbol format. As far as I can tell, Microsoft has an internal tool that rearranges the compiled binary to put the most frequently called code at the front of the binary. The OMAP symbols have something to do with debugging symbols that take into account this postlink step.

The Working Set Tuner (WST) program that comes with the Platform SDK performs a similar optimization except that Microsoft's tool goes into the functions whereas WST stops at the function level. Microsoft's tool looks like it goes down to what is called the basic block level. The arrows in the following code snippet delineate a basic block.

 if ( TRUE == bIsError ) {   <- The basic block starts here.     // Do the error handling here. }   <- The basic block ends here. 

The Microsoft tool moves the error handler to the end of the binary so that only the most common code goes into the front. The OMAP symbols seem to be some sort of fixup to the main symbols because the Microsoft tool manipulated the binary after it was built.

Accessing Symbol Information

To access symbol information, you can use Microsoft's DBGHELP.DLL symbol engine. DBGHELP.DLL can read COFF and C7 symbol formats by itself. To read PDB files, you must also have MSDBI.DLL, which DBGHELP.DLL uses internally. In the past, the symbol engine was in IMAGEHLP.DLL, but Microsoft wisely pulled the symbol engine out of the core system and put it in a DLL that was easier to upgrade. If you have a program that was using the symbol engine when it was part of IMAGEHLP.DLL, IMAGEHLP.DLL still includes the symbol engine exports. The new IMAGEHLP.DLL forwards those functions to DBGHELP.DLL. At the time I was writing this book, the MSDN documentation for the symbol engine was still included as part of IMAGEHLP.DLL.

The DBGHELP.DLL symbol engine allows you to turn an address into the closest public function or global variable. It can also handle the inverse, where you ask it to find the address of a specific function. Finally, you can retrieve the source file and line number for a specific address as well. The DBGHELP.DLL symbol engine doesn't support looking up parameters or local variables, nor does it support type evaluation. As you'll see later in this book, with just this limited functionality, I was able to build some excellent utilities to help you find problems in your applications faster. Microsoft has been slowly improving the symbol table access, and I hope that in the future, DBGHELP.DLL will support our complete symbol table needs.

For WDBG, I used a simple C++ wrapper class, shown in Listing 4-7, that I originally wrote as part of my BUGSLAYERUTIL.DLL library. It is a paper-thin layer of the existing DBGHELP.DLL symbol engine API, but it does provide some workarounds to problems that I've encountered with older IMAGEHLP.DLL symbol engine versions. I left the workarounds in the source code in case you need to use the class with the older IMAGEHLP.DLL symbol engine. I use this class extensively later in the book, so you might want to study it closely.

Listing 4-7 SYMBOLENGINE.H

 /*---------------------------------------------------------------------- "Debugging Applications" (Microsoft Press) Copyright (c) 1997-2000 John Robbins -- All rights reserved. ------------------------------------------------------------------------ This class is a paper-thin layer around the DBGHELP.DLL symbol engine. This class wraps only those functions that take the unique HANDLE value. Other DBGHELP.DLL symbol engine functions are global in scope, so I didn t wrap them with this class. ------------------------------------------------------------------------ Compilation Defines: DO_NOT_WORK_AROUND_SRCLINE_BUG - If defined, the class will NOT work around the SymGetLineFromAddr bug where PDB fMile lookups fail after the first lookup. USE_BUGSLAYERUTIL - If defined, the class will have another initialization method, BSUSymInitialize, which will use BSUSymInitialize from BUGSLAYERUTIL.DLL to initialize the symbol engine and allow the invade process flag to work for all Win32 operating systems. If you use this define, you must use BUGSLAYERUTIL.H to include this file. ----------------------------------------------------------------------*/ #ifndef _SYMBOLENGINE_H #define _SYMBOLENGINE_H // You could include either IMAGEHLP.DLL or DBGHELP.DLL. #include "imagehlp.h" #include <tchar.h> // Include these in case the user forgets to link against them. #pragma comment (lib,"dbghelp.lib") #pragma comment (lib,"version.lib") // The great Bugslayer idea of creating wrapper classes on structures // that have size fields came from fellow MSJ columnist, Paul DiLascia. // Thanks, Paul! // I didn t wrap IMAGEHLP_SYMBOL because that is a variable-size // structure. // The IMAGEHLP_MODULE wrapper class struct CImageHlp_Module : public IMAGEHLP_MODULE { CImageHlp_Module ( ) { memset ( this , NULL , sizeof ( IMAGEHLP_MODULE ) ) ; SizeOfStruct = sizeof ( IMAGEHLP_MODULE ) ; } } ; // The IMAGEHLP_LINE wrapper class struct CImageHlp_Line : public IMAGEHLP_LINE { CImageHlp_Line ( ) { memset ( this , NULL , sizeof ( IMAGEHLP_LINE ) ) ; SizeOfStruct = sizeof ( IMAGEHLP_LINE ) ; } } ; // The symbol engine class class CSymbolEngine { /*---------------------------------------------------------------------- Public Construction and Destruction ----------------------------------------------------------------------*/ public : // To use this class, call the SymInitialize member function to // initialize the symbol engine and then use the other member // functions in place of their corresponding DBGHELP.DLL functions. CSymbolEngine ( void ) { } virtual ~CSymbolEngine ( void ) { } /*---------------------------------------------------------------------- Public Helper Information Functions ----------------------------------------------------------------------*/ public : // Returns the file version of DBGHELP.DLL being used. // To convert the return values into a readable format: // wsprintf ( szVer , // _T ( "%d.%02d.%d.%d" ) , // HIWORD ( dwMS ) , // LOWORD ( dwMS ) , // HIWORD ( dwLS ) , // LOWORD ( dwLS ) ) ; // szVer will contain a string like: 5.00.1878.1 BOOL GetImageHlpVersion ( DWORD & dwMS , DWORD & dwLS ) { return( GetInMemoryFileVersion ( _T ( "DBGHELP.DLL" ) , dwMS , dwLS ) ) ; } BOOL GetDbgHelpVersion ( DWORD & dwMS , DWORD & dwLS ) { return( GetInMemoryFileVersion ( _T ( "DBGHELP.DLL" ) , dwMS , dwLS ) ) ; } // Returns the file version of the PDB reading DLLs BOOL GetPDBReaderVersion ( DWORD & dwMS , DWORD & dwLS ) { // First try MSDBI.DLL. if ( TRUE == GetInMemoryFileVersion ( _T ( "MSDBI.DLL" ) , dwMS , dwLS ) ) { return ( TRUE ) ; } else if ( TRUE == GetInMemoryFileVersion ( _T ( "MSPDB60.DLL" ), dwMS , dwLS ) ) { return ( TRUE ) ; } // Just fall down to MSPDB50.DLL. return ( GetInMemoryFileVersion ( _T ( "MSPDB50.DLL" ) , dwMS , dwLS ) ) ; } // The worker function used by the previous two functions BOOL GetInMemoryFileVersion ( LPCTSTR szFile , DWORD & dwMS , DWORD & dwLS ) { HMODULE hInstIH = GetModuleHandle ( szFile ) ; // Get the full filename of the loaded version. TCHAR szImageHlp[ MAX_PATH ] ; GetModuleFileName ( hInstIH , szImageHlp , MAX_PATH ) ; dwMS = 0 ; dwLS = 0 ; // Get the version information size. DWORD dwVerInfoHandle ; DWORD dwVerSize ; dwVerSize = GetFileVersionInfoSize ( szImageHlp , &dwVerInfoHandle ) ; if ( 0 == dwVerSize ) { return ( FALSE ) ; } // Got the version size, now get the version information. LPVOID lpData = (LPVOID)new TCHAR [ dwVerSize ] ; if ( FALSE == GetFileVersionInfo ( szImageHlp , dwVerInfoHandle , dwVerSize , lpData ) ) { delete [] lpData ; return ( FALSE ) ; } VS_FIXEDFILEINFO * lpVerInfo ; UINT uiLen ; BOOL bRet = VerQueryValue ( lpData , _T ( "\\" ) , (LPVOID*)&lpVerInfo , &uiLen ) ; if ( TRUE == bRet ) { dwMS = lpVerInfo->dwFileVersionMS ; dwLS = lpVerInfo->dwFileVersionLS ; } delete [] lpData ; return ( bRet ) ; } /*---------------------------------------------------------------------- Public Initialization and Cleanup ----------------------------------------------------------------------*/ public : BOOL SymInitialize ( IN HANDLE hProcess , IN LPSTR UserSearchPath , IN BOOL fInvadeProcess ) { m_hProcess = hProcess ; return ( ::SymInitialize ( hProcess , UserSearchPath , fInvadeProcess ) ) ; } #ifdef USE_BUGSLAYERUTIL BOOL BSUSymInitialize ( DWORD dwPID , HANDLE hProcess , PSTR UserSearchPath , BOOL fInvadeProcess ) { m_hProcess = hProcess ; return ( ::BSUSymInitialize ( dwPID , hProcess , UserSearchPath , fInvadeProcess ) ) ; } #endif // USE_BUGSLAYERUTIL BOOL SymCleanup ( void ) { return ( ::SymCleanup ( m_hProcess ) ) ; } /*---------------------------------------------------------------------- Public Module Manipulation ----------------------------------------------------------------------*/ public : BOOL SymEnumerateModules ( IN PSYM_ENUMMODULES_CALLBACK EnumModulesCallback, IN PVOID UserContext ) { return ( ::SymEnumerateModules ( m_hProcess , EnumModulesCallback , UserContext ) ) ; } BOOL SymLoadModule ( IN HANDLE hFile , IN PSTR ImageName , IN PSTR ModuleName , IN DWORD BaseOfDll , IN DWORD SizeOfDll ) { return ( ::SymLoadModule ( m_hProcess , hFile , ImageName , ModuleName , BaseOfDll , SizeOfDll ) ) ; } BOOL EnumerateLoadedModules ( IN PENUMLOADED_MODULES_CALLBACK EnumLoadedModulesCallback, IN PVOID UserContext ) { return ( ::EnumerateLoadedModules ( m_hProcess , EnumLoadedModulesCallback , UserContext ) ); } BOOL SymUnloadModule ( IN DWORD BaseOfDll ) { return ( ::SymUnloadModule ( m_hProcess , BaseOfDll ) ) ; } BOOL SymGetModuleInfo ( IN DWORD dwAddr , OUT PIMAGEHLP_MODULE ModuleInfo ) { return ( ::SymGetModuleInfo ( m_hProcess , dwAddr , ModuleInfo ) ) ; } DWORD SymGetModuleBase ( IN DWORD dwAddr ) { return ( ::SymGetModuleBase ( m_hProcess , dwAddr ) ) ; } /*---------------------------------------------------------------------- Public Symbol Manipulation ----------------------------------------------------------------------*/ public : BOOL SymEnumerateSymbols ( IN DWORD BaseOfDll, IN PSYM_ENUMSYMBOLS_CALLBACK EnumSymbolsCallback, IN PVOID UserContext ) { return ( ::SymEnumerateSymbols ( m_hProcess , BaseOfDll , EnumSymbolsCallback , UserContext ) ) ; } BOOL SymGetSymFromAddr ( IN DWORD dwAddr , OUT PDWORD pdwDisplacement , OUT PIMAGEHLP_SYMBOL Symbol ) { return ( ::SymGetSymFromAddr ( m_hProcess , dwAddr , pdwDisplacement , Symbol ) ) ; } BOOL SymGetSymFromName ( IN LPSTR Name , OUT PIMAGEHLP_SYMBOL Symbol ) { return ( ::SymGetSymFromName ( m_hProcess , Name , Symbol ) ) ; } BOOL SymGetSymNext ( IN OUT PIMAGEHLP_SYMBOL Symbol ) { return ( ::SymGetSymNext ( m_hProcess , Symbol ) ) ; } BOOL SymGetSymPrev ( IN OUT PIMAGEHLP_SYMBOL Symbol ) { return ( ::SymGetSymPrev ( m_hProcess , Symbol ) ) ; } /*---------------------------------------------------------------------- Public Source Line Manipulation ----------------------------------------------------------------------*/ public : BOOL SymGetLineFromAddr ( IN DWORD dwAddr , OUT PDWORD pdwDisplacement , OUT PIMAGEHLP_LINE Line ) { #ifdef DO_NOT_WORK_AROUND_SRCLINE_BUG // Just pass along the values returned by the main function. return ( ::SymGetLineFromAddr ( m_hProcess , dwAddr , pdwDisplacement , Line ) ) ; #else // The problem is that the symbol engine finds only those source // line addresses (after the first lookup) that fall exactly on // a zero displacement. I ll walk backward 100 bytes to // find the line and return the proper displacement. DWORD dwTempDis = 0 ; while ( FALSE == ::SymGetLineFromAddr ( m_hProcess , dwAddr - dwTempDis , pdwDisplacement , Line ) ) { dwTempDis += 1 ; if ( 100 == dwTempDis ) { return ( FALSE ) ; } } // I found it and the source line information is correct, so I ll // change the displacement if I had to search backward to find // the source line. if ( 0 != dwTempDis ) { *pdwDisplacement = dwTempDis ; } return ( TRUE ) ; #endif // DO_NOT_WORK_AROUND_SRCLINE_BUG } BOOL SymGetLineFromName ( IN LPSTR ModuleName , IN LPSTR FileName , IN DWORD dwLineNumber , OUT PLONG plDisplacement , IN OUT PIMAGEHLP_LINE Line ) { return ( ::SymGetLineFromName ( m_hProcess , ModuleName , FileName , dwLineNumber , plDisplacement , Line ) ) ; } BOOL SymGetLineNext ( IN OUT PIMAGEHLP_LINE Line ) { return ( ::SymGetLineNext ( m_hProcess , Line ) ) ; } BOOL SymGetLinePrev ( IN OUT PIMAGEHLP_LINE Line ) { return ( ::SymGetLinePrev ( m_hProcess , Line ) ) ; } BOOL SymMatchFileName ( IN LPSTR FileName , IN LPSTR Match , OUT LPSTR * FileNameStop , OUT LPSTR * MatchStop ) { return ( ::SymMatchFileName ( FileName , Match , FileNameStop , MatchStop ) ) ; } /*---------------------------------------------------------------------- Public Miscellaneous Members ----------------------------------------------------------------------*/ public : LPVOID SymFunctionTableAccess ( DWORD AddrBase ) { return ( ::SymFunctionTableAccess ( m_hProcess , AddrBase ) ) ; } BOOL SymGetSearchPath ( OUT LPSTR SearchPath , IN DWORD SearchPathLength ) { return ( ::SymGetSearchPath ( m_hProcess , SearchPath , SearchPathLength ) ) ; } BOOL SymSetSearchPath ( IN LPSTR SearchPath ) { return ( ::SymSetSearchPath ( m_hProcess , SearchPath ) ) ; } BOOL SymRegisterCallback ( IN PSYMBOL_REGISTERED_CALLBACK CallbackFunction, IN PVOID UserContext ) { return ( ::SymRegisterCallback ( m_hProcess , CallbackFunction , UserContext ) ) ; } /*---------------------------------------------------------------------- Protected Data Members ----------------------------------------------------------------------*/ protected : // The unique value that will be used for this instance of the // symbol engine. This value doesn t have to be an actual // process value, just a unique value. HANDLE m_hProcess ; } ; #endif // _SYMBOLENGINE_H 

Before Windows 2000 became available, getting the Microsoft-supplied symbol engine working was no easy task. The main reason for the difficulty was that the symbol engine was in IMAGEHLP.DLL and many programs used it. Because you can't replace a DLL that is currently loaded, getting a newer version of IMAGEHLP.DLL onto your machine was a challenge. Now that DBGHELP.DLL and MSDBI.DLL aren't system DLLs, they are much easier to upgrade. The most current versions will always be available from the most current Platform SDK, so you should look to install it first. You can download the latest version of the Platform SDK from www.microsoft.com or get it as part of your MSDN subscription service. All the code in this book is set up to use DBGHELP.DLL, so you must have DBGHELP.DLL and MSDBI.DLL on your computer and in a location specified by the PATH environment variable.

Having DBGHELP.DLL installed is only part of the battle, because you need to ensure that you have your symbol files accessible to the symbol engine in order to load them. For DBG files, the DBGHELP.DLL symbol engine will look for them in the following places:

• The current working directory of the application using DBGHELP.DLL, not the debuggee
• The _NT_SYMBOL_PATH environment variable
• The _NT_ALT_SYMBOL_PATH environment variable
• The SYSTEMROOT environment variable

The environment variables must point to directories that are set up a specific way. For example, if your symbols are located in c:\MyFiles, you must create a directory named Symbols under your main directory. Under the Symbols directory, you must create a directory for each extension your binary files use. For example, if you have an EXE and a couple of DLLs, your final directory tree would look like the following. The DBG files for each of your particular extensions go in the appropriate places.

 c:\MyFiles    c:\MyFiles\Symbols    c:\MyFiles\Symbols\Exe    c:\MyFiles\Symbols\Dll 

For PDB files, the only difference is that the DBGHELP.DLL symbol engine will look in the binary for the original PDB path and try to load the PDB from that absolute directory. If the DBGHELP.DLL symbol engine can't load the PDB file from that directory, it will attempt to load the PDB file using the same steps I described previously for DBG files.

Walking the Stack

Fortunately for all of us, we don't have to write our own stack-walking code. DBGHELP.DLL provides the StackWalk API function. StackWalk is straightforward and takes care of all your stack-walking needs. WDBG uses the StackWalk API function just as the Visual C++ debugger does. The only snag you might encounter is that the documentation isn't explicit about what needs to be set in the STACKFRAME structure. Listing 4-8 shows you the exact fields that need to be filled out in the STACKFRAME structure.

StackWalk does such a good job of taking care of the details that you might not be aware that stack walking can be difficult with optimized code. The reason for the difficulty is that the compiler can optimize away the stack frame, the place where the code pushes stack entries, for some functions. The Visual C++ and Visual Basic compilers are aggressive when they do their optimization, and if they can use the stack frame register as a scratch register, they will. To facilitate walking the stack in such situations, the compiler generates what is called Frame Pointer Omission (FPO) data. The FPO data is a table of information that StackWalk uses to figure out how to handle those functions missing a normal stack frame. I wanted to mention FPO because occasionally you'll see references to it on MSDN and in various debuggers. If you're curious, WINNT.H contains the FPO data structures.

Listing 4-8 InitializeStackFrameWithContext from I386CPUHELP.C

 BOOL CPUHELP_DLLINTERFACE __stdcall     InitializeStackFrameWithContext ( STACKFRAME * pStack ,                                       CONTEXT * pCtx ) {     ASSERT ( FALSE == IsBadReadPtr ( pCtx , sizeof ( CONTEXT ) ) ) ;     ASSERT ( FALSE == IsBadWritePtr ( pStack , sizeof ( STACKFRAME ) ) );     if ( ( TRUE == IsBadReadPtr ( pCtx , sizeof ( CONTEXT ) ) ) ||          ( TRUE == IsBadWritePtr ( pStack , sizeof ( STACKFRAME ) ) ) )  {        return ( FALSE ) ;  }   pStack->AddrPC.Offset = pCtx->Eip ; pStack->AddrPC.Mode = AddrModeFlat ; pStack->AddrStack.Offset = pCtx->Esp ; pStack->AddrStack.Mode = AddrModeFlat ; pStack->AddrFrame.Offset = pCtx->Ebp ; pStack->AddrFrame.Mode = AddrModeFlat ; return ( TRUE ) ; } 

Step Into, Step Over, and Step Out

Now that I've described breakpoints and the symbol engine, I want to explain how debuggers implement the excellent Step Into, Step Over, and Step Out functionality. I didn't implement these features in WDBG because I wanted to concentrate on the core portions of the debugger. Step Into, Step Over, and Step Out require source and disassembly views that allow you to keep track of the current executing line or instruction. After you read the discussion in this section, you'll see that the core architecture of WDBG has the infrastructure you need to wire these features in and that adding these features is mostly an exercise in UI programming.

Step Into, Step Over, and Step Out all work with one-shot breakpoints, which, as you'll recall from earlier in the chapter, are breakpoints that the debugger discards after the breakpoints trigger. In the Debug Break discussion earlier in the chapter, you saw another instance in which the debugger uses one-shot breakpoints to stop the processing.

Step Into works differently depending on whether you're debugging at the source level or the disassembly level. When debugging at the source level, the debugger must rely on one-shot breakpoints because a single high-level language line translates into one or more assembly language lines. If you set the CPU into single-step mode, you would be single stepping individual instructions, not the source lines.

At the source level, the debugger knows the source line you're on. When you execute the debugger's Step Into command, the debugger uses the symbol engine to look up the address of the next line to execute. The debugger will do a partial disassembly at the next line address to see whether the line is a call instruction. If the line is a call instruction, the debugger will set a one-shot breakpoint on the first address of the function the debuggee is about to call. If the next line address isn't a call instruction, the debugger sets a one-shot breakpoint there. After setting the one-shot breakpoint, the debugger will release the debuggee so that it runs to the freshly set one-shot breakpoint. When the one-shot breakpoint triggers, the debugger will replace the opcode at the one-shot location and free any memory associated with the one-shot breakpoint. If the user is working at the disassembly level, Step Into is much easier to implement because the debugger will just force the CPU into single-step execution.

Step Over is similar to Step Into in that the debugger must look up the next line in the symbol engine and does the partial disassembly at the line address. The difference is that in Step Over the debugger will set a one-shot breakpoint after the call instruction if the line is a call.

The Step Out operation is in some ways the simplest of the three. When the user selects the Step Out command, the debugger walks the stack to find the return address for the current function and sets a one-shot breakpoint on that address.

The processing for Step Into, Step Over, and Step Out seems straightforward, but there's one small twist that you need to consider. If you write your debugger to handle Step Into, Step Over, and Step Out, what are you going to do if you've set the one-shot breakpoint for those cases and a regular breakpoint triggers before the one-shot breakpoint? As a debugger writer, you have two choices. The first is to leave your one-shot breakpoints alone so that they trigger. The other option is to remove your one-shot breakpoint when the debugger notifies you that a regular breakpoint triggered. The latter option is what the Visual C++ debugger does.

Either way of handling this case is correct, but by removing the one-shot breakpoint for Step Into, Step Over, and Step Out, you avoid user confusion. If you allow the one-shot breakpoint to trigger after the normal breakpoint, the user can easily be left wondering why the debugger stopped at an odd location.

An Interesting Development Problem with WDBG

In general, I didn't have much trouble developing WDBG. However, one problem that proved to be rather interesting did come up. If you run the Visual C++ debugger, the Output window shows you the complete path to the modules as they load. Because I was trying to make WDBG as complete as I could, I wanted to duplicate that functionality. I didn't think doing so would be that difficult.

If you look at the following definition of the LOAD_DLL_DEBUG_INFO structure passed to the debugger on LOAD_DLL_DEBUG_EVENT notifications, you'll see a field for lpImageName, which you would think would be the name of the module loading. That's exactly what it is, but none of the Win32 operating systems ever fills it out.

 typedef struct _LOAD_DLL_DEBUG_INFO {     HANDLE hFile;     LPVOID lpBaseOfDll;     DWORD dwDebugInfoFileOffset;     DWORD nDebugInfoSize;     LPVOID lpImageName;     WORD fUnicode; } LOAD_DLL_DEBUG_INFO;

Because I was loading the module into the DBGHELP.DLL symbol engine as I got the LOAD_DLL_DEBUG_EVENT notifications, I thought I could just look up the complete module name after loading it. The SymGetModuleInfo API function takes an IMAGEHLP_MODULE structure, as shown here, and there is space for the complete module name.

 typedef struct _IMAGEHLP_MODULE {     DWORD SizeOfStruct;     DWORD BaseOfImage;     DWORD ImageSize;     DWORD TimeDateStamp;     DWORD CheckSum;     DWORD NumSyms;     SYM_TYPE SymType;     CHAR ModuleName[32];     CHAR ImageName[256];     CHAR LoadedImageName[256]; } IMAGEHLP_MODULE, *PIMAGEHLP_MODULE;

The puzzling thing I noticed was that SymGetModuleInfo would return that the module symbol information was loaded, but the name of the module would be the name of the DBG symbol file or the module name would be missing completely. This behavior surprised me, but when I thought for a minute, I could see how it might be happening. When I got the LOAD_DLL_DEBUG_INFO structure, the hFile member was valid and I would in turn call SymLoadModule with that hFile. Because I never gave the DBGHELP.DLL symbol engine a full filename to load, it just looked in the open file designated by hFile, found debug information in it, and read in the information. It never needed to know the complete filename.

I just wanted to get the complete name of the module that was loaded. At first, I thought I could use the file handle myself to access the export section of the module and report the module name that I found there. The only problems were that the module could have been renamed so the name in the export section would be wrong, the module could be an EXE or resource-only DLL and not have any exports so it wouldn't have a name, and even if I did retrieve the correct module name, I still wouldn't have the complete path to the module.

After pondering the problem a bit, I figured that there had to be an API function that would take a handle value and tell you the complete name of the open file. When I discovered that there wasn't one, I tried some undocumented means that didn't completely work and that I was hesitant to use in a book like this. I then started poking around with the Tool Help and PSAPI.DLL functions because they will both tell you the modules loaded into a process. The Tool Help functions worked on Windows 98, but on Windows NT 4, PSAPI.DLL failed, and on Windows 2000, the Tool Help functions would hang the debugger hard. The Tool Help functions weren't broken, but they try to start a new thread in the debuggee address space with a call to CreateRemoteThread. Because the debuggee was completely stopped in WDBG, the Tool Help functions would hang until the debuggee restarted. After switching to PSAPI.DLL on Windows 2000, instead of hanging, at least it failed as it did on Windows NT 4.

Using the problem-solving approach that I outlined in Chapter 1, I took stock of the situation and set about formulating some hypothesis to explain the problem. As I read up on the PSAPI.DLL GetModuleFilenameEx function, I started to realize why it might not work when I was calling it. When I was receiving the LOAD_DLL_DEBUG_EVENT notification, it was telling me that a DLL was about to load into the address space, not that the DLL had loaded. Because the memory hadn't been mapped to hold the DLL, the PSAPI.DLL GetModuleFilenameEx was failing; when I stepped through it at the assembly-language level, it appeared to be looking through a mapped memory list that the operating system held for each process.

Now that I knew the source of the problem, I just needed a way to find out when the operating system fully mapped the module into memory. Although I probably could have gone to extreme measures to get this information, such as reverse engineer the image loader in NTDLL.DLL and set a breakpoint there, I opted for a solution that was a little easier and that wouldn't break on each service pack release for the operating system. I figured that I just needed to queue up the module load information and check it every once in a while. PulseModuleNotification is the function I wrote to handle the details of checking the module load information; you can find its implementation in MODULENOTIFICATION.CPP on the companion CD. If you look at the DebugThread function in DEBUGTHREAD.CPP on the companion CD, you'll see that I call PulseModuleNotification every time through the debug loop and whenever the WaitForDebugEvent function times out. Calling PulseModuleNotification continually allows me to report the module load information in a timely manner and with the information that I want.

Common Debugging Question

---

Why can't I step into system functions or set breakpoints in system memory on Windows 98?

If you've ever tried to step into certain system functions on Windows 98, you've seen that the debugger doesn't let you. Windows 2000, on the other hand, allows you to step anywhere you want in your user-mode processes. The reason is that Windows 2000 completely implements copy-on-write, whereas Windows 98 does copy-on-write only for addresses below 2 GB.

As I described in the section "Reading and Writing Memory" earlier in the chapter, copy-on-write allows processes to have their own private copies of mapped memory pages when they, or the debugger, write to a page. Because of the architecture of Windows 98, all processes share the address space above 2 GB. Because Windows 98 doesn't implement copy-on-write for those addresses, if Windows 98 allowed you to set a breakpoint in the shared memory, the first process that executed that address would cause a breakpoint exception. Because that process probably isn't running under a debugger, the process would terminate with a breakpoint exception. Although some system DLLs, such as COMCTL32.DLL, load below 2 GB, the main system DLLs such as KERNEL32.DLL and USER32.DLL load above 2 GB, which means that unless you have a kernel debugger running on Windows 98, you can't step into them with a user-mode debugger.