The Disassembly Window
Now that you've learned some assembly language, the Visual C++ debugger Disassembly window shouldn't be so daunting. The Disassembly window offers many features that will help you with your debugging work. In this section, I'll talk about some of those features and how to minimize the time you spend in the Disassembly window.
Navigating
If you've ever worked with a debugger that didn't have navigation features that let you steer a course through disassembled code, you know that the lack of good navigation tools can lead to a very frustrating debugging experience. Fortunately, the Disassembly window offers several efficient ways to get where you need to go in the debuggee.
The first avenue for getting to a specific location in the debuggee is via the Go To dialog box, which you can reach through Go To on the Edit menu or through an accelerator, Ctrl+G. If you know the address you want to go to, you can just type it in and jump right to it. The Go To dialog box can also interpret symbols and context information, so you can jump to areas even if you don't know the exact address.
The only small problem is that you're stuck with the symbol formatting issues I brought up in Chapter 5. You'll have to do the same translations to account for name decoration that you have to do when setting a breakpoint on a system or an exported function. For example, if you have symbols loaded for KERNEL32.DLL and you want to jump to LoadLibrary in the Disassembly window, you'll need to enter {,,kernel32}_LoadLibraryA@4 in the Go To dialog box to jump to the correct place.
One cool capability that the Disassembly window supports is drag and drop. If you're working through a section of assembly language and you need to quickly check where in memory an operation is going, you can select the address and drag it a couple of pixels. When you release the mouse button, the Disassembly window automatically jumps to that address.
As you're frolicking around the Disassembly window with abandon, don't be surprised if you realize that you've forgotten where you started—it's easy to get lost in the Disassembly window. To get back to where the instruction pointer is sitting, just right-click in the Disassembly window and select Show Next Statement. The Show Next Statement command is also available in source code windows.
You should always use the Go To dialog box to move around in the Disassembly window, especially when you're moving to lower memory. If you try to page or cursor up, you can get into situations in which the Disassembly window needs to disassemble from the beginning of memory to figure out what to display. This complete disassembly can hang the debugger.
Viewing Parameters on the Stack
In Chapter 5, you saw how to set breakpoints on system and exported functions. One of the main reasons for setting breakpoints on these functions is so that you can view the parameters that go into a given function. To demonstrate how to look up items on the stack, I want to use a real-world example instead of a contrived, simple example.
Ever since Visual Basic 5 first came out with the native-code feature, I've wanted to see how the native compilation worked. I saw that the Visual Basic directory included LINK.EXE and C2.EXE. These two programs are also part of Visual C++, and I was curious to see how Visual Basic used them so that I'd have a good idea of how the compilation worked. As you can tell from the name, LINK.EXE links the object files together and produces the executable binary. C2.EXE is a little obtuse. In Visual C++, C2.EXE is the code generator that produces the machine code. I wanted to see whether Visual Basic uses C2.EXE in the same way that Visual C++ does.
From the Visual C++ integrated development environment (IDE), I opened VB6.EXE as the program to debug. Because I had symbols loaded, I needed to set a breakpoint on {,,kernel32}_CreateProcessA@40. When Visual Basic started, I created a simple project, set the project properties to create native code, and selected File_Make from the Visual Basic IDE. The breakpoint on _CreateProcessA@40 gives the debugger control when starting either C2.EXE or LINK.EXE.
On Windows 2000 RC2, the breakpoint on _CreateProcessA@40 stops the debugger at address 0x77E8D7E6 when the instruction about to be executed is PUSH EBP to set up the standard stack frame. Because the breakpoint is on the first instruction of CreateProcess, the top of the stack contains the parameters and the return address. I opened the Memory window by selecting the View_Debug Windows_Memory command. I then entered ESP in the Address field, which is the stack pointer register, to see what was on the stack.
The default format for the Memory window displays all its contents in byte format, and searching through this output can be mind-numbing when you're looking for values that are multiple bytes because you have to do all the Endian conversions in your head. (If you're unfamiliar with the term "Endian," refer to the section "Endians".) One day I right-clicked in the Memory window and, lo and behold, the memory window showed different formats: byte, short hex (2 byte or WORD), and long hex (4 byte or DWORD)
Figure 6-4 shows the stack in the debugger Memory window at the start of the CreateProcess breakpoint. The first value is the return address for the instruction 0xFB6B3F6; the next 10 are the parameters to CreateProcess. (See Table 6-5, below.) CreateProcess has 40 bytes of parameters; each parameter is 4 bytes long. The stack grows from high memory to low memory, and the parameters are pushed in right-to-left order, so the parameters appear in the Memory window in the same order as in the function definition.
Figure 6-4 The stack displayed in the Visual C++ debugger Memory window
You can view the individual parameter values for the first two parameters in two ways. The first way is to use the Memory window by switching its format to byte format and viewing the particular address. The second and easier way is to drag the address you want to view to the Watch window. In the Watch window, use a cast operator to view the address. For example, to view the lpApplicationName parameter in the example, you'd put "(char*)0x0012EAC4" in the Watch window. Either way of viewing works; both should show the following values:
0x0012EAC4 "c:\vb\C2.EXE" 0x0012EBC4 "C@ -il "e:\temp\VB815574 -f "c:\junk\vb\Form1.frm _W 3 _Gy _G5 -Gs4096 _dos _Zl -Fo"c:\junk\vb\Form1.OBJ" _Zi _QIfdiv -ML _basic" |
Table 6-5 Parameters That VB6.EXE Passes to CreateProcess
| Value | Type | Parameter |
|---|---|---|
| 0x0012EAC4 | LPCTSTR | lpApplicationName |
| 0x0012EBC4 | LPTSTR | lpCommandLine |
| 0x00000000 | LPSECURITY_ATTRIBUTES | lpProcessAttributes |
| 0x00000000 | LPSECURITY_ATTRIBUTES | lpThreadAttributes |
| 0x00000001 | BOOL | bInheritHandles |
| 0x08000000 | DWORD | dwCreationFlags |
| 0x00000000 | LPVOID | lpEnvironment |
| 0x00000000 | LPCTSTR | lpCurrentDirectory |
| 0x0012EA3C | LPSTARTUPINFO | lpStartupInfo |
| 0x0012EC60 | LPPROCESS_INFORMATION | lpProcessInformation |
It was easy to get the preceding parameters because I stopped the function on the first instruction before it had a chance to push additional items. If you need to check the parameters when you're in the middle of a function, you have to do a little more work. If you can find the positive offsets from EBP, that helps. Sometimes the best technique is just to open the Memory window and start looking.
The Set Next Statement Command
Like source windows, the Disassembly window has a Set Next Statement command available from the right-click menu, so you can change EIP to another location to execute. You can get away with being a little sloppy with the Set Next Statement line in a source view, but you must be very careful with the Set Next Statement command in the Disassembly window.
The key to getting EIP set right—so that you don't crash—is to pay attention to the stack. Stack pops should balance out stack pushes; if they don't, you'll eventually crash your program. I don't mean to scare you off from changing execution on the fly; in fact, I encourage you to experiment with it. Changing execution with Set Next Statement is a powerful technique and can greatly speed up your debugging. If you take care of the stack, the stack will take care of you.
For example, if you want to reexecute a function without crashing immediately, make sure to change the execution so that the stack stays balanced. Here I want to execute the call to the function at 0x00401005 twice.
00401032 PUSH EBP 00401033 MOV EBP , ESP 00401035 PUSH 404410h 0040103A CALL 00401005h 0040103F ADD ESP , 4 00401042 POP EBP 00401043 RET |
As I step through the disassembly twice, I need to make sure that I let the ADD instruction at address 0x0040103F execute to keep the stack balanced. As the discussion of the different calling conventions earlier in the chapter indicated, the assembly-language snippet shows a call to a __cdecl function because of the ADD instruction right after the call. To reexecute the function, I'd set the instruction pointer to 0x00401035 to ensure that the PUSH occurs properly.
The Memory Window and the Disassembly Window
The Memory window and the Disassembly window have a symbiotic relationship. As you're trying to determine what a sequence of assembly-language operations is doing in the Disassembly window, you need to have the Memory window open so that you can look at the addresses and values being manipulated. Assembly-language instructions work on memory, and memory affects the execution of assembly language; the Disassembly window and the Memory window together allow you to observe the dynamics of this relationship.
On its own, the Memory window is just a sea of numbers, especially when you crash. By combining the two windows, however, you can start figuring out some nasty crash problems. Using these windows together is especially important when you're trying to debug optimized code and the debugger can't walk the stack as easily. To solve the crash, you have to walk the stack manually.
The first step in figuring out how to walk the stack is to know where your binaries are loaded into memory. In version 6, the Visual C++ debugger added a dialog box, Module List, that shows you all the binaries loaded in your program. This dialog box shows you the module name, the path to the module, the load order, and most important, the address range where the module was loaded. Because this dialog box is modal to the debugger, you might want to write down the module names and their load addresses because you'll be referring to this information often. By comparing items on the stack with the list of address ranges, you can get an idea of which items are addresses in your modules.
After you look at the load address ranges, you need to open both the Memory and Disassembly windows. In the Memory window, enter ESP, the stack register, into the Address field and show the values in double-word format by right-clicking within the window and selecting Long Hex Format. Using either your written list of load addresses or the Module List dialog box, start looking at the Memory window numbers across and down. When you see a number that looks as if it belongs to one of your loaded modules, select the entire number and drag it to the Disassembly window. The Disassembly window will show the assembly language at that address, and if you built your application with full debugging information, you should be able to see what the caller function was.
If the ESP register dump doesn't show anything that looks like a module address, you can also dump the EBP register in the Memory window and do the same sorts of lookups. As you get more comfortable with assembly language, you'll start looking at the disassembly around the address that crashed. Studying the crash crime scene should give you some hints about where the return address might be located, either in ESP or in EBP.
The downside to looking up items on the stack manually is that you might have to hunt a considerable way down the stack before you start finding addresses that make sense. If you have an idea of where the modules loaded, however, you should be able to pick out the appropriate addresses quickly.
Let me backtrack a minute before I go any further. Before you use the Memory window, you should know about some of its strengths and weaknesses. On the plus side, the Memory window is the only place where you can view character strings over 255 characters—all other views in the debugger stop at 255. You can also select and drag any variable or memory into the Memory window to view it.
On the downside, the Memory window has two problems. The first problem is that you can view only one piece of memory at a time with the Visual C++ debugger, and there's no way around this limitation. If enough of us ask, however, perhaps future versions of the Visual C++ debugger will allow us to view more than one memory block at a time.
The other problem with the Memory window is that the memory address that you're interested in viewing can bounce around in the display. This bouncing around happens mostly when you right-click to change the memory format. For example, if you position the memory at 0x0012FDBC at the top of the Memory window and then change the memory format, the display can shift down so that the address appears in the middle of the window. As you're concentrating on the problem at hand, you can easily miss the fact that the address you need to see has moved and you start looking at the top line, which is something completely different. I don't know how many times this "jumping memory address" problem has messed up a debugging session.
To work around the memory-address-as-moving-target problem, I've found it best to right-click only on the address I need to see—and nowhere else in the Memory window. Apparently, the Memory window keeps track of where it displays the current address line. For example, if the current address line is the tenth line from the top of the window and you type a new address in the Address field, the new address gets displayed as the tenth line. When you right-click in the window and change the memory format, the Memory window moves the current address line to the right-click location—the result can be extremely confusing.
Debugging War Story
What can go wrong in GlobalUnlock? It's just a pointer dereference.The Battle
A team called me in to help them debug an extremely nasty crash—one severe enough to prevent them from releasing the product. The team had spent nearly a month trying to duplicate the crash and still had no idea what was causing it. The only clue they had was that the crash happened only after they brought up the Print dialog box and changed some settings. After they closed the Print dialog box, the crash occurred a little later in a third-party control. The crash call stack indicated that the crash happened in the middle of GlobalUnlock.
The OutcomeAt first I wasn't sure that anyone was still using the handle-based memory functions (GlobalAlloc, GlobalLock, GlobalFree, and GlobalUnlock) in Win32 programming. After looking at a disassembly of the third-party control, however, I saw that the control writer obviously ported the control from a 16-bit code base. My first hypothesis was that the control wasn't properly dealing with the handle-based memory API functions.
To test my hypothesis, I set some breakpoints on GlobalAlloc, GlobalLock, and GlobalUnlock so that I could find the places in the third-party control where memory was being allocated or manipulated. Once I got the breakpoints set in the third-party control, I started watching how the control used the handle-based memory. Everything seemed normal until I started working through the steps to duplicate the crash.
At some point after closing the Print dialog box, I noticed that GlobalAlloc was starting to return handle values that ended in odd values, such as 5. Because the handle-based memory in Win32 just needs a pointer dereference to turn the handle into a memory value, I immediately saw that I was on to a critical problem. Every memory allocation in Win32 must end in 0, 4, 8, or a C hex digit, because all pointers must be double-word aligned. The handle values coming out of GlobalAlloc were evidence that something serious was corrupted.
Armed with this information, the product manager was ready to jump on the phone and demand the source code for the third-party control because he was sure that the control was causing the crash and holding up his release. After calming him down, I told him that what we had found didn't prove anything and that I needed to be absolutely sure the control was the culprit before we made life miserable for the vendor. I continued to look at the control's memory usage and spent the next several hours chasing down all the handle-based memory manipulations in the control. The control was properly using the handle-based memory, and my new hypothesis became that the team's application contained the real problem. The crash in the third-party control was just a coincidence.
After looking through the team's code, I was more confused than ever because the application was a complete Win32 application and did nothing with handle-based memory. I then turned to their printing code and started looking at it. The code looked fine.
I went back and started to narrow down the minimum case that would duplicate the crash. After a few runs, I found that all I needed to do to crash was to bring up the Print dialog box and change the paper orientation. After closing the Print dialog box, I just needed to reopen it, and the crash would happen shortly after I closed it the second time. I was happy to get the problem duplicated to this minute level because the page orientation was probably just changing a byte somewhere in memory and causing the problem.
Although the code looked fine on the initial read, I went back through each line and double-checked it against the MSDN documentation. After 10 minutes, I found the bug. The team was saving the PRINTDLG data structure used to initialize the Print dialog box with the PrintDlg API function. The third field in the PRINTDLG structure, hDevMode, is a handle-based memory value that the Print dialog box allocates for you. The bug was that the team was using that memory value as a regular pointer and not properly dereferencing the handle or calling GlobalLock on the handle. When they would go to change values in the DEVMODE structure, they were actually writing to the global handle table for the process. The global handle table is a chunk of memory in which all handle-based heap allocations are stored. By having the wild writes to the global handle table, a call to GlobalAlloc would use invalid offsets and values calculated from the global handle table so that GlobalAlloc was returning pointers that were incorrect.
The LessonThe first lesson is to read the documentation carefully. If the documentation says that the data structure is a "movable global memory object," the memory is handle-based memory and you need to dereference that memory handle properly or use GlobalLock on it. Even though 16-bit Windows 3.1 seems like ancient history, some 16-bit-isms are still in the Win32 API, and you must pay attention to them.
Another lesson that I learned was that the global handle table is stored in writable memory. I would have thought that such an important operating system structure would have been in read-only memory. After pondering the reasons Microsoft wouldn't protect that memory, I can only hazard a guess about why they didn't choose to make the global handle table read-only. Technically, the handle-based memory is just for backward compatibility, and Win32 applications should be using the Win32-specific memory types. Protecting the global handle table would require two context switches, from user mode to kernel mode, on each handle-based memory function call. Because those context switches are very expensive in processing time, I can see why Microsoft didn't protect the global handle table.
The final lesson I learned was that I spent too much time concentrating on the third-party control. In all, it took me around seven hours to find the bug. However, the fact that the bug could be duplicated only by bringing up the Print dialog box, which went through the team's code, should have tipped me off that the problem was closer to home.
EAN: 2147483647
Pages: 122