1.3. Specific Features of Windows Programming

This section is a brief introduction to Windows programming. It doesn't pretend to play the role of a learning course; that would require a separate book. I only want to remind you about the main principles of Windows programming, which hopefully will be useful when analyzing executable modules.

1.3.1. General Concepts

Windows programming is based on the use of application program interface (API) functions. Using API functions, an application can communicate directly with the Windows operating system. Applications built on the basis of such interactions are more tightly integrated into the operating system and, consequently, have more powerful capabilities in comparison to other programs. Sometimes, API functions are called system calls. However, this designation is not particularly correct. System calls (in UNIX, for example) are calls to system procedures stored in the operating system kernel. The operating system provides a range of such procedures to simplify resource management for application programs. API functions are an additional interface layer between system procedures and application programs. When calling an API function, you do not know whether it would be executed entirely by the code of the dynamic link library (DLL) loaded into your address space or it would use some procedures stored in the kernel. The Windows operating system is changing and evolving, newer versions are released, but API remains without changes (although new functions might be added to it). Thus, it becomes possible to achieve full compatibility with programs written using only the basic set of API functions.

API functions are supported by using DLLs stored in the system directory (windows\system32). Linking of these libraries is ensured by the compiler (so-called late implicit binding). The total number of API functions is enormous; it exceeds 3,000. Most intensely used are API functions located in the following four DLLs:

Kerne132.dll — This library stores the main system control and management functions (including functions for controlling memory, applications, resources, and files).
User32.dll — This library contains various functions of the user interface (including the ones for processing window messages, timers, and menus).
Gdi32.dll — This is the graphics device interface (GDI) library, containing lots of graphics window functions.
Comctl32.dll — This library contains functions that service various controls. In particular, this library is responsible for the new control style (Windows XP interface style).

If the API function accepts a pointer to string as one of the input arguments, then such a function has two versions: the one with the A prefix for ANSI strings, and one with the W prefix for the Unicode strings. For example, there are two versions for the MessageBox API function: MessageBoxA and MessageBoxW. In high-level programming languages, such as C++, it is necessary to initially determine, with which strings the program operates. Therefore, the compiler automatically selects an appropriate version of the function. When writing a program in Assembly, it is necessary to explicitly specify, which version of a specific function should be used.

There are two main types of application programs under Windows: console applications and graphical user interface (GUI) applications. A specific feature of a console application is that when executing such an application, the system creates a text window, called the console, for this application (or, as a variant, the application inherits the console from the parent process). GUI applications work with graphical windows that can contain graphics and various controls, such as buttons, edit fields, and list boxes. GUI applications are also called graphical or windowing applications. Windows can run other types of applications — services and drivers, which are also known as system applications. In addition, Windows can run applications in Posix and OS/2 subsystems, although with limited possibilities. These types of applications will not be covered in detail in this book.

Usually, Windows programs are written using library functions (C/C++) or library classes (in Delphi, they are called components). In this case, interaction with the operating system is hidden under the layer of libraries. As a result, the analysis of the executable code becomes more complicated because it becomes necessary to determine, which library function or class is in use. This can be achieved by analyzing the library code to determine, which API functions are called, and to understand the aim of these calls. These are not trivial tasks. The goal of this section is to explain the general structure of a Windows program to enable you to understand approaches to analysis of API calls.

In essence, all differences between console and GUI applications consist of the Subsystem flag stored in the portable executable (PE) header (see Section 1.5). This flag is set when linking an application. The following command-line options should be chosen when linking applications using linkexe: /SUBSYSTEM:WINDOWS for GUI applications and /SUBSYSTEM:CONSOLE for console applications. Accordingly, when working with high-level programming languages, the compiler must provide options that allow you to choose between console and GUI applications. At the same time, console and GUI applications are equal in access rights to the operating system resources. Any console application can create graphical windows and work with them, and any GUI application, in turn, can work with console windows.

1.3.2. Console Applications

Console applications are compact, both in compiled form and in source code form. The console itself deserves special attention. As you presumably know, a console is a text-mode window. Interaction between a console application and such a window is reduced to the following:

If a console application is started by another console program, then a child program by default inherits the console of the parent program.
If the parent program has no console, the system creates a new one for the newly-started application.
A console application can have only one console.
A console program can create a new console using the AllocConsole API function, provided that it gets rid of the existing console.

One reason the console appeared in the Windows operating system, which initially was oriented toward graphics applications, was the necessity of running older applications written for MS-DOS. As you may recall, MS-DOS was initially oriented toward working with text strings. When running such a program, Windows automatically allocates a console for it and automatically redirects to the console all its input and output.

The classical structure of the console application can be called a batch structure (Listing 1.4). The program consists of the sequence of the actions that need to be executed. For example, the program might open some file, carry out some actions, and then close the file and terminate operation.

Listing 1.4: ^[5] A typical console application

 #include <windows.h> char *s = "Example of console program.\n\0"; char buf[100]; DWORD d; void main() { // Free the console if it has been inherited.         FreeConsole(); // Create a new console.         AllocConsole(); // Obtain the output handle for console output.         HANDLE ho = GetStdHandle(STD_OUTPUT_HANDLE); // Obtain the handle for console input.         HANDLE hi = GetStdHandle(STD_INPUT_HANDLE); // Output a string to the console.         WriteConsole(ho, s, lstrlen(s), &d, NULL); // Use the ReadConsole function for viewing the console screen.         ReadConsole(hi, (void*)buf, 100, &d, NULL); // Close the handles.         CloseHandle(ho);         CloseHandle(hi); // Free the console.         FreeConsole(); }

Listing 1.4 shows an example of a typical console application that outputs a string to the text screen. A specific feature of this program is that it creates its own console, no matter whether it was started from a console or otherwise. The sequence of FreeConsole()/AllocConsole() function calls frees the existing program console and creates a new one. Nothing happens to the inherited console; the program simply gains the possibility of creating its own console. If you remove the FreeConsole function in the beginning of this program and start it from the console application, then no new console would be created. The program will redirect all of its output to the existing console, despite the presence of the AllocConsole() function.

The program in Listing 1.4 is based on API functions. Even the lstrlen function used for obtaining the string length is actually an API function. Now, consider how IDA Pro^[6] recognizes the executable code (Listing 1.5).

Listing 1.5: The disassembled listing of the executable code

 .text:00401000 _main    proc near           ; CODE XREF: start + 16E↓p .text:00401000          push   ebx .text:00401001          mov    ebx, ds:FreeConsole .text:00401007          push   esi .text:00401008          push   edi .text:00401009          call   ebx                       ; FreeConsole .text:0040100B          call   ds:AllocConsole .text:00401011  mov    edi, ds:GetStdHandle .text:00401017  push   0FFFFFFF5h                   ; nStdHandle .text:00401019  call   edi                          ; GetStdHandle .text:0040101B  push   0FFFFFFF6h                   ; nStdHandle .text:0040101D  mov    esi, eax .text:0040101F  call   edi                          ; GetStdHandle .text:00401021  push   0 ;                          ; lpReserved .text:00401023  mov    edi, eax .text:00401025  mov    eax, lpString .text:0040102A  push   offset NumberOfCharsWritten  ; lpNumberOfCharsWritten .text:0040102F  push   eax                           ; lpString .text:00401030  call   ds:lstrlenA .text:00401036  mov    ecx, lpString .text:0040103C  push   eax                ; nNumberOfCharsToWrite .text:0040103D  push   ecx                          ; lpBuffer .text:0040103E  push   esi                          ; hConsoleOutput .text:0040103F  call   ds:WriteConsoleA .text:00401045  push   0                            ; lpReserved .text:00401047  push   offset NumberOfCharsWritten  ; lpNumberOfCharsRead .text:0040104C  push   64h                 ; nNumberOfCharsToRead .text:0040104E  push   offset unk_4072C8            ; lpBuffer .text:00401053  push   edi                          ; hConsoleInput .text:00401054  call   ds:ReadConsoleA .text:0040105A  push   esi                          ; hObject .text:0040105B  mov    esi, ds:CloseHandle .text:00401061  call   esi                          ; CloseHandle .text:00401063  push   edi                          ; hObject .text:00401064  call   esi                          ; CloseHandle .text:00401066  call   ebx                          ; FreeConsole .text:00401068  pop    edi .text:00401069         pop   esi .text:0040106A         xor   eax, eax .text:0040106C         pop   ebx .text:0040106D         retn .text:0040106D _main   endp

Even at the first glance of an inexperienced user, it becomes immediately clear that the IDA Pro disassembler has solved the problem of disassembling executable code excellently. Nevertheless, in this chapter I am not going to describe disassembled listings; the next and further chapters will concentrate on this problem. For the moment, I would only like to draw your attention to how the programs written using only API functions produce a transparent and clearly understandable executable code.

When speaking about programs similar to the one shown in Listing 1.4, most programmers use C++ library functions instead of API functions. Listing 1.6 represents such a program.

Listing 1.6: An example of a console application using C++ library functions instead of API functions

 #include <stdio.h> char *s = "Example of console program.\n\0"; char buf[100]; void main() {         puts(s);         gets(buf); }

It is necessary to mention that the program in Listing 1.6 doesn't create a new console of its own but uses the console provided by the operating system. In general, however, its features and behavior are the same as the ones of the program shown in Listing 1.4. For working with the console, this program uses the puts and gets functions. The most interesting feature here is that the IDA Pro disassembler easily disassembles standard C++. Looking deeper into the code of the puts function, for example, you can easily notice that execution of this function is finally reduced to execution of the WriteFile API function, which in this case is equivalent to the WriteConsole function. However, application developers often use nonstandard libraries, the functions of which cannot be easily recognized and whose goals are not immediately clear. In particular, this happens if you attempt to disassemble a program written in Delphi. For example, in the Delphi environment execution of the write console operator requires you to call two library procedures, the intention and goals of which cannot be recognized by IDA Pro.

The linear (or, in other words, batch) structure of a console program is simple enough. Although the operations as such might be complex, their sequential order considerably simplifies code investigation. However, if you want to write a program that would tightly interact with the user, you'll have to process keyboard and mouse events. In this case, the program structure would become considerably more complicated. You'll have to introduce a function for processing the main console events and a loop for processing keyboard and mouse events. Listing 1.7 shows an approximate design of such a program.

Listing 1.7: An example of a console program that interacts with the user

 #include <windows.h> BOOL WINAPI handler(DWORD); void inputcons(); void print(char *); HANDLE h1, h2; char *sl  = "Error input!\n"; char s2[35]; char *s4  = "CTRL+C\n"; char *s5  = "CTRL+BREAK\n"; char *s6  = "CLOSE\n"; char *s7  = "LOGOFF\n"; char *s8  = "SHUTDOWN\n"; char *s9  = "CTRL\n"; char *s10 = "ALT\n"; char *s11 = "SHIFT\n"; char *s12 = "\n"; char *s13 = "Code %d \n"; char *s14 = "CAPSLOCK \n"; char *s15 = "NUMLOCK \n"; char *s16 = "SCROLLOCK \n"; char *s17 = "Enhanced key (virtual code) %d \n"; char *s18 = "Function key (virtual code) %d \n"; char *s19 = "Left mouse button\n"; char *s20 = "Right mouse button\n"; char *s21 = "Double click\n"; char *s22 = "Wheel was rolled\n"; char *s23 = "Character '%c' \n"; char *s24 = "Location of cursor x=%d y=%d\n"; void main() { // Console initialization         FreeConsole();         AllocConsole(); // Obtain the output handle.         h1 = GetStdHandle(STD_OUTPUT_HANDLE); // Obtain the input handle.         h2 = GetStdHandle(STD_INPUT_HANDLE); // Set the events handler.         SetConsoleCtrlHandler(handler, TRUE); // Call the function with the message-processing loop.         inputcons(); // Delete the handler.         SetConsoleCtrlHandler(handler, FALSE); // Close the handles.         CloseHandle(h1); CloseHandle(h2); // Free the console.         FreeConsole(); // Exit the program.         ExitProcess(0); }; // Events handler BOOL WINAPI handler(DWORD ct) { //Is this a <CTRL>+<C> event?        if(ct == CTRL_C_EVENT) print(s4); //Is this a <CTRL>+<BREAK> event?        if(ct == CTRL_BREAK_EVENT) print(s5); // Is it necessary to close the console?         if(ct == CTRL_CLOSE_EVENT)         {                  print(s6);                  Sleep(2000);                  ExitProcess(0);         }; // Is it necessary to terminate the session?         if(ct == CTRL_LOGOFF_EVENT)         {                 print(s7);                 Sleep(2000);                 ExitProcess(0);         }; // Is it necessary to terminate the operation?         if(ct == CTRL_SHUTDOWN_EVENT)         {                 print(s8);                 Sleep(2000);                 ExitProcess(0);         };         return TRUE; }; // The function containing the console's message-processing loop void inputcons() {         DWORD n;         INPUT_RECORD ir;         while(ReadConsoleInput(h2, &ir, 1, &n))         { // Process mouse events.                 if(ir.EventType == MOUSE_EVENT)                 { // Double-click.                      if(ir.Event.MouseEvent.dwEventFlags == DOUBLE_CLICK)                                 print(s21); // Move the mouse cursor over the console.                      if(ir.Event.MouseEvent.dwEventFlags == MOUSE_MOVED)                      {                  wsprintf(s2, s24, ir.Event.MouseEvent.dwMousePosition.X,                           ir.Event.MouseEvent.dwMousePosition.Y);                                 print(s2);                         }; // Mouse wheel                     if(ir.Event.MouseEvent.dwEventFlags == MOUSE_WHEELED)                                 print(s22); // Left mouse button if(ir.Event.MouseEvent.dwButtonState == FROM-LEFT_1ST_BUTTON_PRESSED)                 print (s19); // Right mouse button if(ir.Event.MouseEvent.dwButtonState == RIGHTMOST_BUTTON_PRESSED)                 print(s20);                 };                 if(ir.EventType == KEY_EVENT)                 {                         if(ir.Event.KeyEvent.bKeyDown != 1)continue; // Extended keyboard if(ir.Event.KeyEvent.dwControlKeyState == ENHANCED_KEY)                        { wsprintf(s2, s17, ir.Event.KeyEvent.wVirtualKeyCode);                                 print(s2);                        }; //Is this the <CAPS LOCK> key? if(ir.Event.KeyEvent.dwControlKeyState == CRPSLOCK_ON)                                 print(s!4); //Is this the left <ALT> key? if(ir.Event.KeyEvent.dwControlKeyState == LEFT_ALT_PRESSED)                                 print(s10); //Is this the right <ALT> key? if(ir.Event.KeyEvent.dwControlKeyState == RIGHT_ALT_PRESSED)                                 print(s10); //Is this the left <CTRL> key? if(ir.Event.KeyEvent.dwControlKeyState == LEFT_CTRL_PRESSED)                                 print(s9); // Is this the right <CTRL> key? if(ir.Event.KeyEvent.dwControlKeyState == RIGHT_CTRL_PRESSED)                                 print(s9); // Is this the <SHIFT> key? if(ir.Event.KeyEvent.dwControlKeyState == SHIFT_PRESSED)                                 print(s11); //Is this the <NUM LOCK> key? if(ir.Event.KeyEvent.dwControlKeyState == NUMLOCK_ON)                                 print(s15); //Is this the <SCROLL LOCK> key? if(ir.Event.KeyEvent.dwControlKeyState == SCROLLLOCK_ON)                                 print (s16) ; // Handler for normal keys                         if(ir.Event.KeyEvent.uChar.AsciiChar >= 32)                         { wsprintf(s2, s23, ir.Event.KeyEvent.uChar.AsciiChar);                                 print(s2);                         } else                         {                                 if(ir.Event.KeyEvent.uChar.AsciiChar > 0)                                 { // Keys with codes >0 but <32 are processed here. wsprintf(s2, s13, ir.Event.KeyEvent.uChar.AsciiChar);                                   print(s2);                                 } else                                 { // These keys are called functional keys. wsprintf(s2, s 18, ir.Event.KeyEvent.wVirtualKeyCode);                                    print(s2);                                  };                          };                  };        }; // Error message         print(si);         Sleep(5000); }; // Console output function void print(char *s) {         DWORD n;         WriteConsole(h1, s, lstrlen(s), &n, NULL); };

I won't describe this program in detail because I expect that you are an experienced programmer. If you are interested in programming console applications, I recommend that you read my book about Windows programming [3].

When analyzing the program presented in Listing 1.7, it is possible to discover an interesting detail: The handler function is not called explicitly. Its address is specified in the SetConsoleCtrlHandler API function. Naturally, the only method of accessing this important fragment of the program is to analyze the call to the SetConsoleCtrlHandler function to obtain its address. The IDA Pro disassembler behaves in exactly this way. Consider the program fragment shown in Listing 1.8.

Listing 1.8: IDA Pro analyzes the SetConsoleCtrlHandler call to obtain the address of the handler function

 .text:00401453    mov    edi, ds:SetConsoleCtrlHandler .text:00401459    push   1                  ; Add .text:0040145B    push   offset loc_401000  ; HandlerRoutine .text:00401460    mov    hConsoleInput, eax .text:00401465    call   edi                ; SetConsoleCtrlHandler

The disassembler not only correctly recognizes the call to SetConsoleCtrlHandler but also correctly interprets both parameters of this function. Do not become confused by the mov hConsoleInput, eax command; it has no relation to the ConsoleCtrlHandler call. On the contrary, it relates to the previous call — GetStdHandle. This is the cost of optimization.

Note

It should be admitted that contemporary compilers can optimize the code much better than professional programmers in Assembly. A programmer is always bound to observe various conventions, such as programming style and code readability. These conventions do not matter for the compiler. Various methods of optimization will be covered later in this book.

Recall the previously described fragment. Because of the SetConsoleCtrlHandler function, the disassembler correctly determines the beginning of the handler function, which allows correct disassembling of this function.

Pay attention to the inputcons function. Principally, it doesn't contain anything unusual. The calls to the ReadConsoleInput function in the loop allow you to detect events that cannot be traced by the handler function. This loop could be called the message-processing loop of a console application. Such loops are more typical of windowing applications; however, for console applications this approach is also permitted. Naturally, there is a considerable difference between the two methods of message processing. Each application can have only one console window; therefore, there is no need to determine, to which window an individual message relates. A GUI application can have several windows but only one message loop (see Section 1.3.3). Under these circumstances, each message is marked by the handle of the window, for which that message is intended. At this point, some difficulties might arise, which will be explained in the next section.

1.3.3. Windowing Applications

Windowing applications, also known as GUI applications, are based on event-driven mechanisms. In other words, the main part of the code of such applications is concentrated in specialized functions, which, similar to the handler function from the previous section, are called by the system at a specific event. In addition, for this type of application, the presence of the message-processing loop is typical. The message-processing loop is used to redirect each newly-arrived message to the appropriate handler function (Listing 1.9).

Listing 1.9: A typical GUI application

 #include <windows.h> LRESULT CALLBACK WndProc(HWND, UINT, WPARfM, LPflRAM); int APIENTRY WinMain(HINSTRNCE hlnstance,                      HINSTANCE hPrevInstance,                      LPSTR     IpCmdLine,                      int       nCmdShow) {         char cname[] = "Class";         char title[] = "A simple Windows application";         MSG msg;         // The structure for window class registration         WNDCLASS wc;         wc.style         = 0;         we.lpfnWndProc   = (WNDPROC)WndProc;         wc.cbClsExtra    = 0;         wc.cbWndExtra    = 0;         wc.hInstance     = hInstance;         wc.hIcon         = LoadIcon(hInstance, (LPCTSTR)IDI_APPLICATION);         wc.hCursor       = LoadCursor (NULL, IDC_ARROW);         wc.hbrBackground = (HBRUSH)(COLOR_WINDOW + 1) ;         wc.lpszMenuName  = 0;         wc.lpszClassName =cname;         // Register the class.         if(!RegisterClass(&wc)) return 0;         // Create the window.         HWND hWnd = CreateWindow(                cname,                     // Class                title,                     // Header                WS_OVERLAPPEDWINDOW,       // Window style                0,                         // X coordinate                0,                         // Y coordinate                500,                       // Window width                300,                       // Window height                NULL,                      // Handle of the parent window                NULL,                      // Menu handle                hInstance,                 // Application identifier                NULL);                     // Pointer to the structure sent                                           // by the WM_CREATE message         // Check whether the window has been created.         if (!hWnd) return 0;         // Display the new window.         ShowWindow(hWnd, nCmdShow);         // Update the window contents.         UpdateWindow(hWnd);         // Message-processing loop         while (GetMessage(&msg, NULL, 0, 0))         {         // Translate the virtual key codes to ASCII codes.                 TranslateMessage(&msg);         // Redirect the message to the window procedure.                 DispatchMessage(&msg);         }         return 0; }; // Window procedure LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) {         switch(message)         { // Message sent when creating the window         case WM_CREATE:                 break; // Message sent when destroying the window         case WM_DESTROY: // Message sent when exiting the message-processing loop                 PostQuitMessage(0);                 break; // Message sent when redrawing the window contents         case WM_PAINT:                 break; // Return unprocessed messages.                 default:         return DefWindowProc (hWnd, message, wParam, lParam);         }         return 0; }

Listing 1.9 demonstrates a minimal GUI application characterized by all minimal functional capabilities of a Windows application. In general, Windows applications are built on the basis of the main window. All remaining windows "orbit" the main window like planets of the solar system around the sun. Thus, it is easy to distinguish the three main components of such an application:

Definition and registration of the window class, to which the main window should belong
Message-processing loop, the main task of which is "catching" the messages arriving to the application and redirecting them to the required window function (not just the main window function)
The main window function and, possibly, functions of other windows

Being aware of such relationship patterns, it is possible to purposefully search for individual elements of a GUI application.

DispatchMessage is the main API function in the message-processing loop. This function redirects the newly-arrived messages to the given window function. The message structure appears as shown in Listing 1.10.

Listing 1.10: The message structure

 typedef struct {     HWND hwnd;     UINT message;     WPARAM wParam;     LPARAM 1Param;     DWORD time;     POINT pt; } MSG

In the preceding listing, you will find the following:

hwnd — The handle of the window, to which the current message is addressed.
message — The code of the current message.
wParam — An optional parameter containing supplementary information.
lParam — An optional parameter containing supplementary information.
time — The time, at which the message was sent.
pt — The mouse cursor coordinate at the time the message was sent. The least significant word designates the X coordinate, and the most significant word designates the Y coordinate.

The hwnd value defines the window, to which the message must be sent. For each window — to be more precise, for each window class — there is a special message-processing function (see Listing 1.9). The system knows this, and the message arrives where necessary. Users, however, do not know this. By the way, the main part of the program code is either concentrated within such functions or is called from them. How is it possible to solve this problem? To solve it (at least, to begin solving it correctly), recall that most window functions must be registered. The function and the window class are registered. For example, consider Listing 1.9: The address of the message-processing function is loaded into the lpfnWndProc field. In other words, having looked at the disassembled application code, you'll learn the function address. For example, consider a fragment of the disassembled listing produced by the IDA Pro disassembler (Listing 1.11).

Listing 1.11: A fragment of a disassembled GUI application code produced by IDA Pro

 .text:00401077 mov [esp + 80h + WndClass.lpfnWndProc], offset loc_401000

Here, loc_401000 determines the window function address. The program understands the RegisterClass function and the structure that this function accepts as an argument. Listing 1.12 shows a fragment obtained using the W32Dasm v. 10 disassembler, which also has a good reputation.

Listing 1.12: A disassembled fragment obtained using the W32Dasm v. 10 disassembler

 :00401077 C744241800104000                 mov [esp+18], 00401000 :0040107F 896C241C                         mov dword ptr [esp+1C], ebp :00401083 896C2420                         mov dword ptr [esp+20], ebp :00401087 89742424                         mov dword ptr [esp+24], esi * Reference To: USER32.LoadIconA, Ord:01BDh                                            | :0040108B FF15C4504000                     Call dword ptr [004050C4] :00401091 68007F0000                       push 00007FOO :00401096 55                               push ebp :00401097 89442428                         mov dword ptr [esp+28], eax * Reference To: USER32.LoadCursorA, Ord:01B9h                                            | :0040109B FF15C8504000                     Call dword ptr [004050C8] :004010A1 89442424                         mov dword ptr [esp+24], eax :004010A5 8D44240C                         lea eax, dword ptr [esp+OC] :004010A9 8D542450                         lea edx, dword ptr [esp+50] :004010AD 50                               push eax :004010AE C744242C06000000                 mov [esp+2C], 00000006 :004010B6 896C2430                         mov dword ptr [esp+30], ebp :004010BA 89542434                         mov dword ptr [esp+34], edx * Reference To: USER32.RegisterClassA, Ord:0216h                                   | :004010BE FF15CC504000                Call dword ptr [004050CC] :004010C4 6685C0                           test ax, ax

Having carefully considered the listing produced by W32Dasm, you should immediately conclude that it is considerably less informative than the one generated by IDA Pro. Nevertheless, in most cases, it correctly determines API functions. Thus, it is easy to find the RegisterClass function. Then, by the other functions preceding RegisterClass, it is possible to conclude that the mov [esp + 18], 00401000 command assigns the value of the window function address to the lpfnWndProc field. Thus, having detected the window function, an investigator can analyze its text and then find an individual fragment that carries out the specific action.

The window function is intended for processing messages delivered to it. There are lots of messages informing the window function about various events that occur to the window or some of its controls. Finally, it is possible to send custom, user-defined messages to the window function. For this purpose, there is a special WM_USER constant, and all messages defined programmatically must be greater than or equal to this constant. By the text of the window function, it is possible to determine the reaction of the program to a specific event and thus to understand the working mechanism of the specific GUI application.

The problem, however, is that the window function doesn't relate to a specific window: It relates to the entire class of windows. When the application is based on API programming, one function can correspond to one window. However, this is rarely the case. Processing messages intended for different windows doesn't require considerable effort, because every message contains a window descriptor (handle). However, this results in certain difficulties when analyzing executable code, because in the course of static code analysis it is difficult to determine, for which window the message processed by the current code fragment is intended. At this point, debuggers are helpful; they can help set breakpoints to the code of the window function or, as with the Softlce debugger, even to a specific message from a specific window.

Naturally, the message-processing loop plays an extremely important role in every GUI program. Having located it in the disassembled code, you'd be able to locate the program fragment that precedes the loop — in other words, determine where in the program the main window is created and where the main window class is registered. To search for the message-processing loop, use such API functions as GetMessage, PeekMessage, TranslateMessage, and DispatcheMessage, as well as the IsDialogMessage function.

1.3.4. Applications Based on Dialogs

Listing 1.13 presents an example of an application, in which the main window is a modal dialog (Fig. 1.4).

Listing 1.13: An example application that uses a modal dialog

 // Resource identifiers // Definitions of style constants #define WS_VISIBLE             0x0l0000000L #define WS_SYSMENU             0x000S0000L #define WS_MINIMIZEBOX         0x00020000L #define WS_MAXIMIZEBOX         0x000l0000L // Modal dialog definition DIALOG DIALOGEX 10, 10, 150, 100 STYLE  WS_VISIBLE | WS_SYSMENU | WS_MINIMIZEBOX |WS_MAXIMIZEBOX CAPTION "Modal dialog" FONT 12, "Arial" { } // Program module #include <windows.h> int DWndProc(HWND, UINT, WPARAM, LPARAM); __stdcall WinMain(HINSTANCE hInstance,     HINSTANCE hPrevInstance,     LPSTR lpCmdLine,     int nCmdShow ) { // Create a modal dialog. DialogBoxParam(hInstance, "DIALOG", NULL, (DLGPROC)DWndProc, 0); // Close the application. ExitProcess(0); }; // Message-handling function of the modal dialog int DWndProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM 1Param)         {                switch(uMsg)         { // Message that arrives when the dialog is created         case WM_INITDIALOG:                break; // Message that arrives in case of an attempt at closing the window         case WM_CLOSE:         EndDialog(hwndDlg, 0);         return TRUE; // Message from window controls         case WM_COMMAND:                 break;         };                 return FALSE;         };

image from book
Figure 1.4: An example of a dialog (Listing 1.13)

In contrast to normal windows, modal dialogs are characterized by the following features:

Modal dialogs are created on the basis of a template stored in program resources or created in the memory. In the example from Listing 1.13, the modal dialog is created on the basis of the template stored in the resources file.
Modal dialogs are created using the DialogBoxParam function. The fourth parameter of this function specifies the address of the function that processes window messages. The DialogBoxParam function doesn't return control until the EndDialog function is called.
The message-processing function of a dialog is similar to the message-processing function of a normal window. If the function receives the message and processes that message itself, it returns TRUE; otherwise, it returns FALSE. As relates to messages, the main difference is that the WM_INITDIALOG message comes to the dialog instead of the WM_CREATE message that comes to a normal window.
In contrast to a normal window, the dialog has no message-processing loop. To be more precise, there is one, but the operating system creates it and processes and redirects the message. Thus, you can encounter applications that have no obvious message-processing loops.
Important issues of working with modal dialogs are the processing of the WM_CLOSE message and the call to the EndDialog function, which removes the modal dialog from the memory.

By the way, the window called by the MessageBox API function is a typical example of a modal dialog box. In this case, the system not only processes the message but also creates the window template and organizes the window message function.

Note

In the resources file (see Listing 1.13), window style constants are defined explicitly. However, this is not necessary. You can simply insert the following line of code: #include windows.h>. Alternatively, you can use the Resource Wizard of the Visual Studio .NET product, after which you needn't worry about the contents of the resources file.

Again, consider how IDA Pro disassembled this file (Listing 1.14). The DialogBoxParam function helps you find the dialog's message-processing function.

Listing 1.14: The disassembled code of the fragment shown in Listing 1.13

 .text:00401000 ; BOOL __stdcall DialogFunc(HWND, UINT, WPARAM, LPARAM) .text:00401000 DialogFunc proc near ; DATA XREF: WinMain(x, x, x, x) + 6↓o .text:00401000 .text:00401000            hDlg  = dword ptr 4 .text:00401000            arg_4 = dword ptr 8 .text:00401000 .text:00401000            cmp  [esp+arg_4], 10h .text:00401005            jnz  short loc_401014 .text:00401007            mov  eax, [esp + hDlg] .text:0040100B            push 0          ; nResult .text:0040100D            push eax        ; hDlg .text:0040100E            call ds:EndDialog .text:00401014 .text:00401014 loc_401014:                ; CODE XREF: DialogFunc + 5|j .text:00401014            xor  eax, eax .text:00401016            retn .text:00401016 DialogFunc endp .text:00401016 .text:00401016 ; ----------------------------------------------------- .text:00401017                align 10h .text:00401020 .text:00401020 ; ------- S U B R O U T I N E -------------------------- .text:00401020 .text:00401020 .text:00401020 ; __stdcall WinMain(x,x,x,x) .text:00401020 _WinMain@16 proc near        ; CODE XREF: start + 186↓p .text:00401020 .text:00401020 hInstance   = dword ptr 4 .text:00401020 .text:00401020             mov     eax, [esp + hInstance] .text:00401024             push    0                   ; dwInitParam .text:00401026             push    offset DialogFunc   ; lpDialogFunc .text:0040102B             push    0                   ; hWndParent .text:0040102D             push    offset TemplateName ; lpTemplateName .text:00401032             push    eax                 ; hInstance .text:00401033             call    ds:DialogBoxParamA  ; Create a modal .text:00401033                           ; dialog box from a dialog box. .text:00401033                                   ; Template resource .text:00401039                push    0                   ; uExitCode .text:0040103B                call    ds:ExitProcess .text:00401041                int     3           ; Trap to debugger. .text:00401041 _WinMain@16    endp

Finally, it is necessary to mention another type of window: nonmodal dialogs. Windows of this type require an explicit message-processing loop. Listing 1.15 provides an example of an application, in which a nonmodal dialog plays the role of the main window.

Listing 1.15: An example application, in which a nonmodal dialog plays the role of the main window

 // Resources file // Resource identifiers // Definitions of style constants #define WS_VISIBLE                  0x0l0000000L #define WS_SYSMENU                  0x00080000L #define WS_MINIMIZEBOX              0x00020000L #define WS_MAXIMIZEBOX              0x000l0000L // Definition of the nonmodal dialog DIALOG DIALOGEX 10, 10, 150, 100 STYLE  WS_VISIBLE | WS_SYSMENU | WS_MINIMIZEBOX |WS_MAXIMIZEBOX CAPTION "Nonmodal dialog" FONT 12, "Arial" { } // Program module #include <windows.h> MSG msg; int DWndProc(HWND, UINT, WPARAM, LPARAM); __stdcall WinMain(HINSTANCE hInstance,     HINSTANCE hPrevInstance,     LPSTR lpCmdLine,     int nCmdShow) { // Nonmodal dialog     HWND hdlg = CreateDialog(hInstance, "DIALOG", NULL, (DLGPROC)DWndProc); // Message-processing loop         while (GetMessage(&msg, NULL, 0, 0))         {                 IsDialogMessage(hdlg, &msg);         } // Close the application.        ExitProcess(0); }; // Window function of the nonmodal dialog int DWndProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)         {         switch(uMsg)         { // Message coming when the dialog is created         case WM_INITDIALOG:                 break; // Message coming in an attempt at closing the dialog         case WM_DESTROY:                 PostQuitMessage(0);                 break;         case WM_CLOSE:                 DestroyWindow (hwndDlg);                 return TRUE; // Message from window controls         case WM_COMMAND:                 break;         };                 return FALSE;         };

As you can see from Listing 1.15, the program is similar to a normal windowing application. However, there still are some specific features:

The most obvious feature is that there is no window class registration block.
The message-processing loop is slightly modified. Instead of the normal TranslateMessage and DispatchMessage functions, the IsDialogMessage function is used. The use of the latter relates to the problem with using the <Tab> key for switching among the window controls. The IsDialogMessage function is used to ensure that everything is working correctly in the nonmodal dialog. In general, when an application contains both normal windows and nonmodal dialogs, the message-processing loop might appear as shown in Listing 1.16.

Listing 1.16: Message-processing loop of an application containing normal windows and nonmodal dialogs
```
 while (GetMessage(&msg, NULL, 0, 0)) { if(!IsDialogMessage(hw, &msg))            {                TranslateMessage(&msg);                DispatchMessage(&msg);            } } 
```

Here, hw is the nonmodal dialog handle. However, the IsDialogMessage function can also be used for a normal window.

A certain difference in processing of the window closing event (clicking the Close button in the top right corner) also attracts attention. A normal window is actually closed by the system and, accordingly, the WM_DESTROY message is delivered to the window function, which is processed to exit the message-processing loop (PostQuitMessage). The nonmodal window is not closed automatically; therefore, it is necessary to process the WM_CLOSE message and then close the window using the DestroyWindow function. There are no secrets here. The DefwindowProc function processes the WM_CLOSE message and implicitly calls the DestroyWindow function.

^[5]All C++ programs in this book were developed in the Visual Studio .NET environment unless stated otherwise.

^[6]The IDA Pro 4.7 disassembler will be used for listings throughout this book.