List of Listings

Chapter 1: Introduction to Disassembling

Listing 1.1: A simple program that outputs the memory dump

Listing 1.2: A fragment of a program written in C

Listing 1.3: A sequence of different variables

Listing 1.4: A typical console application

Listing 1.5: The disassembled listing of the executable code

Listing 1.6: An example of a console application using C++ library functions instead of API functions

Listing 1.7: An example of a console program that interacts with the user

Listing 1.8: IDA Pro analyzes the SetConsoleCtrlHandler call to obtain the address of the handler function

Listing 1.9: A typical GUI application

Listing 1.10: The message structure

Listing 1.11: A fragment of a disassembled GUI application code produced by IDA Pro

Listing 1.12: A disassembled fragment obtained using the W32Dasm v. 10 disassembler

Listing 1.13: An example application that uses a modal dialog

Listing 1.14: The disassembled code of the fragment shown in Listing 1.13

Listing 1.15: An example application, in which a nonmodal dialog plays the role of the main window

Listing 1.16: Message-processing loop of an application containing normal windows and nonmodal dialogs

Listing 1.17: A small Assembly program for studying formats of the Intel processor commands

Listing 1.18: A fragment of the Assembly program intended for studying the jump instructions format

Listing 1.19: Investigating conditional jump codes when the offset in a 32-bit segment acts as the address

Listing 1.20: The dump of the code presented in Listing 1.19

Listing 1.21: A test program for studying the format of MOV commands

Listing 1.22: The disassembled listing of the program shown in Listing 1.21

Listing 1.23: A fragment of the test program for studying the role of the SIB byte

Listing 1.24: The machine code corresponding to the fragment shown in Listing 1.23

Listing 1.25: The commands decoded using the manual disassembling technique

Listing 1.26: The command sequence clarifying that some 32-bit variable is located at the 4176A8H address

Listing 1.27: A test program illustrating different methods of calling procedures

Listing 1.28: The standard MS-DOS stub

Listing 1.29: The IMAGE_DOS_HEADER structure

Listing 1.30: A simple program for determining whether this file is a loadable PE module

Listing 1.31: The IMAGE_NT_HEADERS structure

Listing 1.32: The IMAGE_FILE_HEADER structure

Listing 1.33: The IMAGE_OPTIONAL_HEADER32 structure

Listing 1.34: The IMAGE_DATA_DIRECTORY structure

Listing 1.35: An elementary structure that makes up a typical element of the sections table

Listing 1.36: The C++ function for determining the object offset in the PE file by its relative virtual address

Listing 1.37: The array of structures in the beginning of the import table

Listing 1.38: The IMAGE_THUNK_DATA32 structure

Listing 1.39: The IMAGE_EXPORT_DIRECTORY structure

Listing 1.40: The IMAGE_RESOURCE_DIRECTORY structure

Listing 1.41: The IMAGE_RESOURCE_DIRECTORY_ENTRY structure

Listing 1.42: Fragment of the winuser.h file

Listing 1.43: An elementary Assembly program

Listing 1.44: The result of disassembling the prog.exe program using the dumpbin.exe utility

Listing 1.45: The contents of the .data section of the test example in Listing 1.44

Listing 1.46: The results produced by hiew.exe when disassembling the prog.exe test program

Listing 1.47: The disassembled text of the prog.exe module compiled and linked using TASM32

Listing 1.48: The test Assembly program for illustrating difficulties with disassembling

Listing 1.49: The modified code of the test program shown in Listing 1.48

Listing 1.50: A fragment of the disassembled text produced by IDA Pro

Listing 1.51: A fragment of the disassembled listing produced by W32Dasm

Listing 1.52: An example demonstrating nonstandard use of the RET command

Listing 1.53: A fragment of code demonstrating another imitation of the JMP command

Listing 1.54: A sequence of commands, for which it is hard to guess where the jump takes place

Listing 1.55: An example illustrating the code overlapping technique

Listing 1.56: A simple console application intended for investigating the code self-modification problem

Listing 1.57: A program that copies the contents of PROC1 into the stack and tries to run it there

Listing 1.58: The stack location, to which the code of the PROC1 procedure has been copied

Listing 1.59: A modified version of the program presented in Listing 1.57

Listing 1.60: The final version that copies the procedure code into the stack and executes it there

Listing 1.61: An example of a self-modifying program that uses the WriteProcessMemory function

Listing 1.62: An example of self-modifying code that uses the VirtualProtectEx command

Chapter 2: The Code Investigator's Toolkit

Listing 2.1: Disassembled listing of the executable code produced by the dumpbin.exe utility

Listing 2.2: The result of execution of the dumpbin /relocations prog.dll command

Listing 2.3: A simple console application that displays a text string

Listing 2.4: Disassembled code of the simple console program provided in Listing 2.3

Listing 2.5: Code lines directly preceding the call displaying the warning message

Listing 2.6: Fragment of the Allscreen code required for removing the delay procedure

Listing 2.7: Fragment of Allscreen responsible for limiting the number of runs

Listing 2.8: Commands you must correct to remove the limitation on program runs

Listing 2.9: Strings in the Intel C compiler console output, which were found in IDA Pro

Listing 2.10: Tracing the console output of the evaluation copy of the Intel C compiler

Listing 2.11: The fragment of disassembled code containing the Register Successfully! string

Listing 2.12: The code fragment, to which the cross-reference (Listing 2.11) points

Listing 2.13: The "suspicious" code located slightly above the fragment shown in Listing 2.12

Listing 2.14: Code fragment located at the loc_4181A4 address of the GetPixel program

Listing 2.15: The GetPixel code fragment that contains a reference to "How do you feel me?"

Listing 2.16: The GetPixel fragment for the Yes/No choice when the nag screen is displayed

Listing 2.17: A candidate for the role of the password-checking procedure

Listing 2.18: Another candidate for the role of the password-checking procedure

Listing 2.19: The code fragment that must be replaced with the MOV EAX, 1 command

Chapter 3: Main Paradigms of the Executable Code Analysis

Listing 3.1: Simple C++ program containing three global variables, one uninitialized

Listing 3.2: Disassembled code of the program (Listing 3.1) compiled without optimization

Listing 3.3: Disassembled code (Listing 3.1) compiled with the "create fast code" option

Listing 3.4: Sample program, in which one global variable is defined using a pointer

Listing 3.5: Disassembled code of the program presented in Listing 3.4

Listing 3.6: Distinction between an address of a global variable and a normal constant

Listing 3.7: Disassembled code of the program shown in Listing 3.6

Listing 3.8: Fragment of the test C program for studying variable size, location, and type

Listing 3.9: Fragment of Listing 3.8, in which variables are declared in a different order

Listing 3.10: Simple example illustrating optimal alignment of data of different sizes

Listing 3.11: Complete codes of commands assigning values to three types of variables

Listing 3.12: Simple program for investigating the behavior of floating-point variables

Listing 3.13: Disassembled code of the main function from Listing 3.12

Listing 3.14: Disassembled code of the executable module produced by Borland C++

Listing 3.15: Simple example illustrating the use of Unicode strings

Listing 3.16: Disassembled listing of the call to the wprintf function

Listing 3.17: Unicode strings from Listing 3.15 as they appear in the data section

Listing 3.18: Example illustrating the use of Delphi strings

Listing 3.19: Code fragment containing strings from the program in Listing 3.18

Listing 3.20: Fragment of the disassembled test program (Listing 3.18) with the s3 string

Listing 3.21: Simple C program illustrating string operations

Listing 3.22: Disassembled code of the program shown in Listing 3.21

Listing 3.23: Disassembled code of the modified program (Listing 3.21)

Listing 3.24: Memory block passed to the stack

Listing 3.25: Simple C program for investigating array identification in the executable code

Listing 3.26: Disassembled code of the program shown in Listing 3.25

Listing 3.27: Sample program for investigating the behavior of structures

Listing 3.28: Disassembled text of the program shown in Listing 3.27

Listing 3.29: Behavior of the structure passed to some function as a parameter

Listing 3.30: Disassembled text of the main function of the program shown in Listing 3.29

Listing 3.31: Disassembled text of the init function (Listing 3.29)

Listing 3.32: Fragment calling Init (compiled by Borland C++ and disassembled by IDA Pro)

Listing 3.33: Disassembled code showing the call to the RegisterClass API function

Listing 3.34: Example program illustrating the use of local variables

Listing 3.35: Disassembled text of the main function from Listing 3.34

Listing 3.36: Disassembled text of the add function (Listing 3.34)

Listing 3.37: Disassembled code of the optimized main function

Listing 3.38: Disassembled main function from Listing 3.34 compiled using Borland C++ 5.0

Listing 3.39: Use of temporary variables on the example of numeric computations

Listing 3.40: Disassembled code of the main function of the program in Listing 3.39

Listing 3.41: Temporary variables when the result of executing a function is used in another

Listing 3.42: Disassembled code of Listing 3.41 for processing intermediate variables

Listing 3.43: Example program illustrating the use of register variables

Listing 3.44: Disassembled code of the program shown in Listing 3.43

Listing 3.45: Algorithm for passing parameters through the program code

Listing 3.46: Simple program illustrating the use of the__fastcall calling convention

Listing 3.47: Disassembled text (Listing 3.46) produced by Microsoft Visual C++

Listing 3.48: Disassembled code (Listing 3.46) compiled using the Borland C++ compiler

Listing 3.49: Pascal program with nested procedures

Listing 3.50: Disassembled code of the program shown in Listing 3.49, compiled using Delphi

Listing 3.51: Disassembled text of the compiled proc1 procedure

Listing 3.52: Disassembled code of the proc2 procedure from Listing 3.49

Listing 3.53: Sequence of commands typical for exiting in the middle of a procedure

Listing 3.54: C++ program illustrating the procedure of identifying function start and end

Listing 3.55: Disassembled code of the main function from Listing 3.54

Listing 3.56: Disassembled code of the myfunc function from Listing 3.54

Listing 3.57: Example of a simple program vulnerable to stack overflow

Listing 3.58: Disassembled code of the main function from Listing 3.57

Listing 3.59: Disassembled code of the getpassword function (Listing 3.57)

Listing 3.60: Executable code that can be placed into the vulnerable buffer (Listing 3.57)

Listing 3.61: Executable code that would operate correctly in the vulnerable buffer

Listing 3.62: Simple conditional construct

Listing 3.63: Disassembled code of the program shown in Listing 3.62

Listing 3.64: Implementation of a complete conditional structure in the executable code

Listing 3.65: Alternative implementation of complete conditional constructs

Listing 3.66: Typical C++ construct

Listing 3.67: Executable code generated using Microsoft Visual C++ (Listing 3.66)

Listing 3.68: Sample program illustrating comparison of real numbers

Listing 3.69: Disassembled code of the program in Listing 3.68 created by IDA Pro

Listing 3.70: Sequence of commands that compares two 8-byte floating-point numbers

Listing 3.71: Example program containing nested conditional constructs

Listing 3.72: Method of nested conditional structures built by the compiler

Listing 3.73: Example illustrating recognition of logical operators

Listing 3.74: Code fragment illustrating the use of the conditional bit setting commands

Listing 3.75: Typical example of using the choice operator

Listing 3.76: Disassembled listing (Listing 3.74) compiled using Microsoft Visual C++

Listing 3.77: Alternative and more elegant method for checking conditions

Listing 3.78: Processing the typical choice operator

Listing 3.79: Typical loop structure

Listing 3.80: Typical example of a postcondition loop

Listing 3.81: Simple example of using the FOR loop

Listing 3.82: Disassembled text compiled using Microsoft Visual C++ without optimization

Listing 3.83: Disassembled text (Listing 3.81) compiled using the "compact code" option

Listing 3.84: Disassembled text (Listing 3.81) compiled using the Borland C++ compiler

Listing 3.85: Simple demonstration of computation at compile time optimization

Listing 3.86: Disassembled text of the program in Listing 3.85

Listing 3.87: Test program demonstrating the loop unwinding technique

Listing 3.88: Disassembled text of the executable code of the program in Listing 3.87

Listing 3.89: Computing array elements

Listing 3.90: Loop implemented by the compiler

Listing 3.91: Simultaneous complex loop exit condition and auxiliary loop control operators

Listing 3.92: Commands for loop exit

Listing 3.93: Commands whose role in the loop structure needs to be clarified

Listing 3.94: Disassembled code that fills a two-dimensional array

Listing 3.95: Simple program demonstrating typical program structures for serving objects

Listing 3.96: Disassembled code of the main function from Listing 3.95

Listing 3.97: Disassembled code (Listing 3.96) corresponding to seta/geta calls (Listing 3.95)

Listing 3.98: Example illustrating the use of dynamic objects

Listing 3.99: Disassembled text of the program shown in Listing 3.98

Listing 3.100: Example program demonstrating a typical example of inheritance

Listing 3.101: Disassembled code of the program compiled by Microsoft Visual C++ without optimization

Listing 3.102: Fragment of the sub_401050 function (Listing 3.101)

Listing 3.103: Program for studying the concepts of virtual functions and polymorphism

Listing 3.104: Disassembled code of the main function from Listing 3.103

Listing 3.105: Mechanism used for forming the virtual function address

Listing 3.106: Disassembled code of the sub_4010A0 function

Listing 3.107: Defining and placing the address of the virtual functions table

Listing 3.108: Virtual functions table in a sophisticated class hierarchy

Listing 3.109: Code of the seta virtual function defined in class B

Listing 3.110: When assigning the value, it must be incremented by one (Listing 3.109)

Listing 3.111: Practical example illustrating the use of constructors and destructors

Listing 3.112: Disassembled code of the main function from Listing 3.111

Listing 3.113: Disassembled code of the constructor (Listing 3.111)

Listing 3.114: Writing the virtual functions table address into the new object instance

Listing 3.115: Code fragment demonstrating the code of the constructor

Listing 3.116: Disassembled code of the procedure calling the destructor (Listing 3.111)

Listing 3.117: Sample program containing a simple numeric computation

Listing 3.118: Disassembled code of the program shown in Listing 3.117

Listing 3.119: Code pattern using lea for optimization of mathematical computations

Listing 3.120: Sample program illustrating the exception handling mechanism

Listing 3.121: Disassembled code of the program presented in Listing 3.119

Listing 3.122: __try block nesting level and exception handling structures

Listing 3.123: Customary sequence that precedes the start of the _except block

Listing 3.124: Standard prototype of the main function of a console application in C++

Listing 3.125: Typical main function call from executable code from Microsoft Visual C++

Listing 3.126: Prototype of the WinMain function

Listing 3.127: Typical WinMain call from the executable module from Microsoft Visual C++

Chapter 4: The SoftIce Debugger

Listing 4.1: Several lines from the Softlce code window when mixed representation was used

Listing 4.2: Information about the ADD processor instruction displayed by the OPINFO command

Chapter 5: The IDA Pro Disassembler

Listing 5.1: Easy Assembly program (see Listing 1.43)

Listing 5.2: Fragment of the program in Listing 5.1 that hides a small trick

Listing 5.3: Disassembled listing of the program in Listing 5.1, produced by IDA Pro

Listing 5.4: Modified code of the program shown in Listing 5.1

Listing 5.5: Disassembled text of the program shown in Listing 5.4

Listing 5.6: Code example, for which IDA Pro correctly identifies the procedure address

Listing 5.7: Simple Assembly program illustrating interactive work of the code investigator and IDA Pro

Listing 5.8: Disassembled code of the program shown in Listing 5.7

Listing 5.9: Function automatically created by IDA Pro

Listing 5.10: Code fragment illustrating automatic generation of cross-references

Listing 5.11: Structure of the IDC function

Listing 5.12: Fragment of an IDC program illustrating conversion of a string variable to an integer type

Listing 5.13: Fragment of an IDC program illustrating conversion of an integer to a string type

Listing 5.14: IDC program fragment converting floating-point numbers to the string data type

Listing 5.15: IDC program fragment illustrating the concatenation operation

Listing 5.16: Example IDC program illustrating memory access and navigation

Listing 5.17: Simple IDC program that illustrating memory reading

Listing 5.18: Output of the program presented in Listing 5.17

Listing 5.19: Simple IDC program that analyzes the specified memory block and patches some bytes

Listing 5.20: Simple IDC program that outputs the Assembly code in the specified address range

Listing 5.21: Result of executing the program shown in Listing 5.20

Listing 5.22: Data types and flags of the attributes byte as defined in the idc.idc file

Listing 5.23: IDC program that outputs to the console addresses of data items and their lengths and types

Listing 5.24: Results of executing the program in Listing 5.23

Listing 5.25: All possible flag elements and values for byte 1 of a data item or instruction (idc.idc file)

Listing 5.26: Program that views the IDA Pro listing and finds software labels for console output

Listing 5.27: Fragment of the idc.idc file containing the list of flags defining the function properties

Listing 5.28: Outputting function names within the address interval; setting library function comments

Listing 5.29: Locating three sequential PUSH commands and moving the cursor to that group

Listing 5.30: Outputting names and identifiers of all structures and enumerations detected by IDA Pro

Appendix 1: A Program for Investigating the PE Header

Listing A1: Sample program demonstrating methods of working with PE module structures

Listing A2: Example of output of the program in Listing A1