In Chapter 5, I presented some tips and tricks that can make your life in the debugger easier. In this chapter, I'll continue to provide you with suggestions, but here I'll concentrate on tips that will help you with debugging at the assembly-language level.
The "Endianness" of a CPU refers to which end of a byte is stored first. Intel CPUs are "Little Endian," which means that the little end of a multibyte value is stored first. For example, the value 0x1234 is stored in memory as 0x34 0x12. It's important that you keep the Little Endian storage in mind when you're looking at memory in the debugger. You'll need to convert it in your head so that you're interpreting the correct values. If you use the Memory window to look at one of your link list nodes and the next pointer value is 0x12345678, the value will be displayed in byte format as 0x78 0x56 0x34 0x12.
If you're curious, the term "Endian" comes from Jonathan Swift's Gulliver's Travels, and the computer meaning came from a 1980 Request for Comments (RFC) concerning byte ordering by Danny Cohen. Danny's paper is at http://www.op.net/docs/RFCs/ien-137 for those who want to know the entire story.
As a crash dumps you into the Disassembly window, you have to determine whether or not you're looking at real code, which is sometimes a difficult task. Here are some tips that will help you figure out whether you're looking at something other than executable code.
The Visual C++ debugger Watch window knows how to decode all the registers to values. Therefore, you can put a register in the Watch window and cast it to the type you want to observe. For example, if you're looking at a string manipulation instruction, you can enter (char*)@EDI in the Watch window to view the data in a format that's easier to read.
If you'd like to see more mixed assembly language and source code, you can have Visual C++ generate the assembly listings for your source files. If you type the /FAs option into the Project Settings dialog box, C++ tab, Project Options edit control, the compiler will generate an ASM file for each of your source code files. You might not want to generate the ASM files for every build, but they can be instructive, letting you see what the compiler is generating. The ASM files don't require you to fire up your application every time you're curious about assembly language.
The files generated are nearly ready to compile with the Microsoft Macro Assembler (MASM), so they can be a challenge to read. Much of the files consist of MASM directives, but the main parts of the files show your C code with the assembly-language code below each construct. After reading this chapter, you shouldn't have any trouble following the flow of ASM files.