Implementing DeadlockDetection
As you can see, using DeadlockDetection is fairly simple. Beneath the simplicity of use, however, is a fairly sophisticated implementation. The first part of DeadlockDetection's implementation I'll go over is how to hook functions.
Hooking Imported Functions
There are many ways to hook the function calls a program makes. The hard way is to hunt down all the CALL instructions and replace the address with one of your own. This approach is difficult and extremely error prone. Fortunately, with DeadlockDetection, the functions I need to hook are imported functions and are much easier to handle than CALL instructions.
An imported function is a function that comes out of a DLL. For example, when your program calls OutputDebugString, it's calling a function that resides in KERNEL32.DLL. When I first started doing Microsoft Win32-based programming, I thought that calling an imported function would be just like calling any other function—a CALL or branch instruction would jump to an address and start executing the imported function. The only difference might be that with an imported function the operating system program loader would have to run through the executable and fix up the addresses to reflect where the DLL being called would be loaded into memory. When I looked at how a call to an imported function really is implemented, I was amazed at the simplicity and beauty of the design.
The problem with the way I was thinking becomes apparent when you consider how many API functions there are and that you can easily call the same ones many times throughout your program. If the loader had to find and replace each occurrence of a call to OutputDebugString, for example, loading a program could take forever. Even if the linker generated a complete table that specified where each call to OutputDebugString took place in the code, the huge amount of looping and memory writing would make load times excruciatingly slow.
So how does the loader tell your application where to find an imported function? The solution is fiendishly clever. If you think about where the calls to OutputDebugString go, you'll soon realize that each call must go to the same address: the address where OutputDebugString is loaded into memory. Of course, your application can't know this address ahead of time, so instead, all your OutputDebugString calls get routed through a single, indirect address. When the program loader loads your executable and its dependent DLLs, the loader fixes up this one indirect address so that it corresponds to the final load address of OutputDebugString. The compiler makes this indirect addressing work by generating a jump to the indirect address any time your code calls the imported function. This indirect address is stored in the .idata (or import) section of the executable. If you import through __declspec(dllimport), instead of being an indirect jump, the code is an indirect call, thus saving a couple of instructions per function call.
Hooking an imported function is a matter of finding the imports section of the executable, looking for the address of the particular function you want to hook, and then writing the hook function address in its place. Although hunting down and replacing function addresses might sound like a lot of work, it's not that bad because the Win32 Portable Executable (PE) file format is so nicely organized.
In Chapter 10 of his excellent book Windows 95 System Programming Secrets (IDG Books, 1995), Matt Pietrek describes a method for hooking imported functions. Matt's code simply finds a module's imports section and, using the value returned from a call to GetProcAddress, loops through the imported functions. When Matt's code finds the function it's looking for, it overwrites the original imported function address with the hook function.
Not surprisingly, given that 1995 is a past lifetime in software development circles, two small issues have changed since Matt's book came out. The first is that when Matt wrote his book, most developers didn't merge their imports section with other PE sections. Therefore, if the imports section is in read-only memory, you cause an access violation by writing the hook address. I fixed the read-only memory issue by setting the virtual memory protection to read-write before the hook function address is written to memory. The second issue, which is a little tougher to work around, is that under certain conditions you can't hook imported functions under Windows 98.
When you use DeadlockDetection, you want to be able to have the threading functions redirected any time you run your application, even when the application is running under the debugger. Although you wouldn't think that hooking functions while running under a debugger would be a problem, it is. In Microsoft Windows 2000, or when running a program in Windows 98 outside a debugger, when you call GetProcAddress to find a function and then look through the imports section for that address, you'll always find it. But under Windows 98, calling GetProcAddress in your program while it's running under a debugger returns a different address than when it runs outside a debugger. What GetProcAddress actually returns when running under the debugger is a debug thunk—a special wrapper around the real call.
Windows 98 doesn't implement copy-on-write in the operating system, as I explained in Chapter 4. The debug thunk returned when running under a debugger is a means by which Windows 98 keeps debuggers from attempting to step into system functions above the 2-GB line. Overall, the lack of copy-on-write isn't much of an issue for most developers—only those who write debuggers or who want to hook functions correctly whether or not they're running under a debugger.
Fortunately, getting the real address for an imported function isn't too difficult—it just takes a little more work, and you have to avoid GetProcAddress. The PE file IMAGE_IMPORT_DESCRIPTOR structure, which holds all the information about functions imported from a specific DLL, has pointers to two arrays in the executable. These arrays are called import address tables (IATs), or sometimes thunk data arrays. The first pointer references the real IAT, which the program loader fixes up when the executable is loaded. The second pointer references the original IAT, which is untouched by the loader and lists the imported functions. To find the real imported function address, simply work your way through the original IAT until you find the named function that you want to hook, and then write the hook address in the corresponding entry in the real IAT, which the program is using. By taking this extra step, the hooking code will always work no matter where it's called.
Listing 12-2 shows HookImportedFunctionsByName, the function I wrote to take care of your hooking needs. Table 12-3 shows the parameters to HookImportedFunctionsByName and describes each one. Because I wanted to make the hooking as generic as possible, I went to the trouble of allowing you to hook multiple functions imported from the same DLL at the same time. As its name implies, the HookImportedFunctionsByName function will hook only those functions imported by name. In Chapter 14, I'll discuss how to hook functions imported by ordinal value as part of the LIMODS utility.
Table 12-3 HookImportedFunctionsByName Parameter Descriptions
| Parameter | Description |
|---|---|
| hModule | The module in which the imports will be hooked. |
| szImportMod | The name of the module whose functions are imported. |
| Count | The number of functions to hook. This parameter is the size of the paHookArray and paOrigFuncs arrays. |
| paHookArray | The array of function descriptor structures that list which functions to hook. The array doesn't have to be in szFunc name order (though it's wise to keep the array sorted in function name order, because I might implement better searching in the future). Also, if a particular pProc is NULL, HookImportedFunctionsByName skips that item. The structure for each element in paHookArray is simply the name of the function to hook and a pointer to the new hook procedure. Because you might want to hook or unhook functions at will, HookImportedFunctionsByName returns all the original imported function addresses. |
| paOrigFuncs | The array of original addresses hooked by HookImportedFunctionsByName. If a function wasn't hooked, that item index will be NULL. |
| pdwHooked | Returns the number of functions hooked out of paHookArray. |
Listing 12-2 HookImportedFunctionsByName from HOOKIMPORTEDFUNCTIONSBYNAME.CPP
BOOL BUGSUTIL_DLLINTERFACE __stdcall HookImportedFunctionsByName ( HMODULE hModule , LPCSTR szImportMod , UINT uiCount , LPHOOKFUNCDESCA paHookArray , PROC * paOrigFuncs , LPDWORD pdwHooked ) { // Assert the parameters. ASSERT ( FALSE == IsBadReadPtr ( hModule , sizeof ( IMAGE_DOS_HEADER ) ) ) ; ASSERT ( FALSE == IsBadStringPtr ( szImportMod , MAX_PATH ) ) ; ASSERT ( 0 != uiCount ) ; ASSERT ( NULL != paHookArray ) ; ASSERT ( FALSE == IsBadReadPtr ( paHookArray , sizeof (HOOKFUNCDESC) * uiCount )); // In debug builds, perform deep validation of paHookArray. #ifdef _DEBUG if ( NULL != paOrigFuncs ) { ASSERT ( FALSE == IsBadWritePtr ( paOrigFuncs , sizeof ( PROC ) * uiCount ) ); } if ( NULL != pdwHooked ) { ASSERT ( FALSE == IsBadWritePtr ( pdwHooked , sizeof ( UINT ))); } // Check each item in the hook array. { for ( UINT i = 0 ; i < uiCount ; i++ ) { ASSERT ( NULL != paHookArray[ i ].szFunc ) ; ASSERT ( '\0' != *paHookArray[ i ].szFunc ) ; // If the function address isn't NULL, it is validated. if ( NULL != paHookArray[ i ].pProc ) { ASSERT ( FALSE == IsBadCodePtr ( paHookArray[i].pProc)); } } } #endif // Perform the error checking for the parameters. if ( ( 0 == uiCount ) || ( NULL == szImportMod ) || ( TRUE == IsBadReadPtr ( paHookArray , sizeof ( HOOKFUNCDESC ) * uiCount ) )) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } if ( ( NULL != paOrigFuncs ) && ( TRUE == IsBadWritePtr ( paOrigFuncs , sizeof ( PROC ) * uiCount ) ) ) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } if ( ( NULL != pdwHooked ) && ( TRUE == IsBadWritePtr ( pdwHooked , sizeof ( UINT ) ) ) ) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } // Is this a system DLL above the 2-GB line, which Windows 98 won't // let you patch? if ( ( FALSE == IsNT ( ) ) && ( (DWORD)hModule >= 0x80000000 ) ) { SetLastErrorEx ( ERROR_INVALID_HANDLE , SLE_ERROR ) ; return ( FALSE ) ; } // TODO TODO // Should each item in the hook array be checked in release builds? if ( NULL != paOrigFuncs ) { // Set all the values in paOrigFuncs to NULL. memset ( paOrigFuncs , NULL , sizeof ( PROC ) * uiCount ) ; } if ( NULL != pdwHooked ) { // Set the number of functions hooked to 0. *pdwHooked = 0 ; } // Get the specific import descriptor. PIMAGE_IMPORT_DESCRIPTOR pImportDesc = GetNamedImportDescriptor ( hModule , szImportMod ); if ( NULL == pImportDesc ) { // The requested module wasn't imported. Don't return an error. return ( TRUE ) ; } // Get the original thunk information for this DLL. I can't use // the thunk information stored in pImportDesc->FirstThunk // because the loader has already changed that array to fix up // all the imports. The original thunk gives me access to the // function names. PIMAGE_THUNK_DATA pOrigThunk = MakePtr ( PIMAGE_THUNK_DATA , hModule , pImportDesc->OriginalFirstThunk ) ; // Get the array the pImportDesc->FirstThunk points to because // I'll do the actual bashing and hooking there. PIMAGE_THUNK_DATA pRealThunk = MakePtr ( PIMAGE_THUNK_DATA , hModule , pImportDesc->FirstThunk ); // Loop through and find the functions to hook. while ( NULL != pOrigThunk->u1.Function ) { // Look only at functions that are imported by name, not those // that are imported by ordinal value. if ( IMAGE_ORDINAL_FLAG != ( pOrigThunk->u1.Ordinal & IMAGE_ORDINAL_FLAG )) { // Look at the name of this imported function. PIMAGE_IMPORT_BY_NAME pByName ; pByName = MakePtr ( PIMAGE_IMPORT_BY_NAME , hModule , pOrigThunk->u1.AddressOfData ) ; // If the name starts with NULL, skip it. if ( '\0' == pByName->Name[ 0 ] ) { continue ; } // Determines whether I hook the function BOOL bDoHook = FALSE ; // TODO TODO // Might want to consider bsearch here. // See whether the imported function name is in the hook // array. Consider requiring paHookArray to be sorted by // function name so that bsearch can be used, which // will make the lookup faster. However, the size of // uiCount coming into this function should be rather // small, so it's OK to search the entire paHookArray for // each function imported by szImportMod. for ( UINT i = 0 ; i < uiCount ; i++ ) { if ( ( paHookArray[i].szFunc[0] == pByName->Name[0] ) && ( 0 == strcmpi ( paHookArray[i].szFunc , (char*)pByName->Name ) ) ) { // If the function address is NULL, exit now; // otherwise, go ahead and hook the function. if ( NULL != paHookArray[ i ].pProc ) { bDoHook = TRUE ; } break ; } } if ( TRUE == bDoHook ) { // I found a function to hook. Now I need to change // the memory protection to writable before I overwrite // the function pointer. Note that I'm now writing into // the real thunk area! MEMORY_BASIC_INFORMATION mbi_thunk ; VirtualQuery ( pRealThunk , &mbi_thunk , sizeof ( MEMORY_BASIC_INFORMATION ) ) ; if ( FALSE == VirtualProtect ( mbi_thunk.BaseAddress , mbi_thunk.RegionSize , PAGE_READWRITE , &mbi_thunk.Protect )) { ASSERT ( !"VirtualProtect failed!" ) ; SetLastErrorEx ( ERROR_INVALID_HANDLE , SLE_ERROR ); return ( FALSE ) ; } // Save the original address if requested. if ( NULL != paOrigFuncs ) { paOrigFuncs[i] = (PROC)pRealThunk->u1.Function ; } // Microsoft has two different definitions of the // PIMAGE_THUNK_DATA fields as they are moving to // support Win64. The W2K RC2 Platform SDK is the // latest header, so I'll use that one and force the // Visual C++ 6 Service Pack 3 headers to deal with it. // Hook the function. DWORD * pTemp = (DWORD*)&pRealThunk->u1.Function ; *pTemp = (DWORD)(paHookArray[i].pProc); DWORD dwOldProtect ; // Change the protection back to what it was before I // overwrote the function pointer. VERIFY ( VirtualProtect ( mbi_thunk.BaseAddress , mbi_thunk.RegionSize , mbi_thunk.Protect , &dwOldProtect ) ) ; if ( NULL != pdwHooked ) { // Increment the total number of functions hooked. *pdwHooked += 1 ; } } } // Increment both tables. pOrigThunk++ ; pRealThunk++ ; } // All OK, Jumpmaster! SetLastError ( ERROR_SUCCESS ) ; return ( TRUE ) ; } |
HookImportedFunctionsByName shouldn't be difficult for you to follow. After doing the proactive debugging practice of validating every parameter thoroughly, I call the helper function GetNamedImportDescriptor to find the IMAGE_IMPORT_DESCRIPTOR for the requested module. After getting the pointers to the original and real IATs, I loop through the original IAT looking at each function imported by name to see whether it's in the paHookArray list. If the function is in the hook list, I simply set the real IAT's memory to PAGE_READWRITE so that the hooked address can be written safely, write the hook into the entry for the real function, and reset the memory protection to its original setting. The unit test function for HookImportedFunctionsByName is included with the BUGSLAYERUTIL.DLL source code on the companion CD, so feel free to use it to step through the code if you don't quite follow what's going on.
Now that you have an idea of how to hook the imported functions, let's move on to implementing the rest of DeadlockDetection.
Implementation Highlights
One of my primary goals in implementing DeadlockDetection was to make the utility as data-driven and table-driven as possible. When you step back and look at what the DLL does in its hook processing, you'll see that the processing is almost identical for each function in Table 12-1. The hooked function gets called, determines whether its class of functions is being monitored, calls the real function, and (if logging is on for that class) logs the information and returns. I had to write a bunch of similar hook functions, and I wanted to make them as simple as possible. Complicated hook functions are a perfect breeding ground for bugs.
The best way to show this simplicity is to talk about writing a DeadDetExt DLL. A DeadDetExt DLL must have three exported functions. The first two, DeadDetExtOpen and DeadDetExtClose, are self-explanatory. The interesting function is DeadDetProcessEvent, which each hook function calls when there is information to write. DeadDetProcessEvent takes a single parameter, a pointer to a DDEVENTINFO structure:
typedef struct tagDDEVENTINFO { // The identifier that specifies what the rest of // this structure contains eFuncEnum eFunc ; // The pre-call or post-call indicator ePrePostEnum ePrePost ; // The return address. This address helps in finding the calling // function. DWORD dwAddr ; // The thread ID of the calling thread DWORD dwThreadId ; // The return value for post calls. DWORD dwRetValue ; // The parameter information. Cast this information to the appropriate // structure, as described below, for the function. When accessing // the parameters, treat them as read-only. DWORD dwParams ; } DDEVENTINFO , * LPDDEVENTINFO ; |
The entire output for a single function that appears in Listing 12-1 comes from the information in the DDEVENTINFO structure. Although most of the fields in DDEVENTINFO are self-explanatory, the dwParams field needs special mention. The dwParams field is really a pointer to the parameters as they appear in memory.
In Chapter 6, I discussed how parameters are passed on the stack. Just to jog your memory, parameters for __stdcall functions are passed right to left and the stack grows from high memory to low memory. The dwParams field in the DDEVENTINFO structure points to the last parameter on the stack, so the structure lists the parameters in left-to-right order. I applied a little creative casting to make it easy to convert dwParams.
In DEADLOCKDETECTION.H, I provide typedefs that describe each intercepted function's parameter lists. For example, if eFunc were eWaitForSingleObjectEx, you would cast dwParams to LPWAITFORSINGLEOBJECTEX_PARAMS to get the parameters. To see all of this creative casting in action, check out the TEXTFILEDDEXT.DLL code included on the companion CD.
Although output processing is relatively easy, gathering the information can be difficult. I wanted DeadlockDetection to hook the synchronization functions in Table 12-1, but I didn't want the hook functions to change the behavior of the real functions. I also wanted to get the parameters and the return value and to write the hook functions in C/C++ easily. I spent quite a while with the debugger and the disassembler before I got it right.
Initially, I wrote all the hook functions so that they were just pass-through functions and called the real functions directly. This approach worked great. Then I put the parameters and the return value for the functions into local variables. Although getting the value returned from the real function was simple, I realized that I didn't have a clean way to get the return address with my C/C++ hook function. I needed the DWORD right before the current stack pointer. Unfortunately, in straight C/C++, the function prolog would've already done its magic by the time I could get control, so the stack pointer would've already moved away from where it needed to be.
You might think that the stack pointer is just offset by the number of local variables, but that isn't always the case. The Visual C++ compiler does a pretty good job of optimizing so that the stack pointer isn't in the same place with different optimization flags set. Although you might declare a variable as a local variable, the compiler can optimize the variable by storing it in a register so that it doesn't even appear on the stack.
I needed a guaranteed way to get the stack pointer no matter what optimizations were set. At this point, I started thinking naked (no, not me without clothes): why not declare the hook functions as __declspec(naked) and create my own prolog and epilog code? With this approach, I'd have complete control over ESP no matter what optimization settings were used. Additionally, getting the return address and parameters would be a snap because they are at ESP+04h and ESP+08h, respectively. Keep in mind that I'm not doing anything out of the ordinary with the prolog and epilog code, so I still perform the usual PUSH EBP and MOV EBP, ESP for the prolog and MOV ESP, EBP and POP EBP for the epilog.
Because each hook function was going to be declared as __declspec(naked), I wrote a couple of macros to handle the prolog and epilog: HOOKFN_PROLOG and HOOKFN_EPILOG. I also went ahead and declared some common local variables that all hook functions would need in HOOKFN_PROLOG. These variables included the last error value, dwLastError, and the event information structure to pass to the DeadDetExt DLL, stEvtInfo. The dwLastError is yet another bit of state that I needed to preserve when intercepting functions.
The Windows API can return a special error code through SetLastError to provide more information if a function fails. This error code can be a real boon because it tells you why an API function failed. For example, if GetLastError returns 122, you know that the buffer parameter was too small. WINERROR.H contains all the error codes the operating system returns. The problem with hook functions is that they can reset the last error as part of their processing. This behavior can wreak havoc if your application relies on the last error.
If you call CreateEvent and want to see whether the returned handle was created or just opened, CreateEvent sets the last error to ERROR_ALREADY_EXISTS if it just opened the handle. Because the cardinal rule of intercepting functions is that you can't change the expected behavior of the function, I needed to call GetLastError immediately after the real function call so that my hook function could properly set the last error code that the real function returned. The general rule for a hook function is that you need to call GetLastError right after you call the real function and then call SetLastError as the last action before leaving the hook function.
At this point, I thought I was done except for testing. Unfortunately, my first test uncovered a bug: I wasn't preserving ESI and EDI across the hook call because the documentation on using the inline assembler explicitly stated that you didn't have to save them. After I fixed the ESI/EDI register problem, DeadlockDetection seemed to work fine. When I started doing register comparisons on before, during, and after cases, however, I noticed that I wasn't returning the values the real functions left in EBX, ECX, and EDX, and worse yet, in the flags registers. Although I didn't see any problems and the documentation said that those registers didn't need to be preserved, I still was concerned that my hook functions were changing the application state. I declared the REGSTATE structure to hold the register values after the real function call so that I could restore them when my hook function returned. To save and restore the registers, I created two additional macros, REAL_FUNC_PRE_CALL and REAL_FUNC_POST_CALL, which I placed around the real call the hook function makes.
After a little more testing, I found another problem: in release builds with full optimizations, I crashed inexplicably every so often. I finally tracked down those crashes and was able to attribute them to the effect of the optimizer on some of my hook functions. The optimizer was trying to be helpful but ended up doing more harm than good. I was very careful about the register usage in my hook functions and used only EAX or stack memory directly. Even though I was taking every precaution to preserve the registers, I found that the debug build code sequence
MOV DWORD PTR [EBP-018h] , 00000002h MOV DWORD PTR [EBP-014h] , 00000002h |
was being transformed by the optimizer into
PUSH 002h POP EBX MOV DWORD PTR [EBP-01Ch] , EBX MOV DWORD PTR [EBP-018h] , EBX |
It's easy to see in the second snippet that the POP into EBX was trashing the register. To prevent the optimizer from stealing registers behind my back, I turned optimizations off for all hook function files by placing a
#pragma optimize("", off ) |
Listing 12-3 shows the final version of DD_FUNCS.H, which is the internal header file in which all the special hook function macros are declared. The comment at the top of the file has a sample hook function that explains how I used each of the special macros. I strongly encourage you to step through the SimpTest example that's part of the source code. Make sure that you watch an entire function call at the assembly-language level because that's the only way you'll see all the processing that takes place.
Listing 12-3 DD_FUNCS.H
/*---------------------------------------------------------------------- "Debugging Applications" (Microsoft Press) Copyright (c) 1997-2000 John Robbins -- All rights reserved. ------------------------------------------------------------------------ The prototypes for all the hook functions and the prolog/epilog code ----------------------------------------------------------------------*/ #ifndef _DD_FUNCS_H #define _DD_FUNCS_H /*////////////////////////////////////////////////////////////////////// All the hook functions are __declspec(naked) functions, so I must provide the prolog and epilog. I need to provide a custom prolog and epilog for several reasons: 1. Functions written in C have no control over which registers are used or when the compiler saves the original registers. Not having control over the registers means that getting the return address is nearly impossible. For the DeadlockDetection project, the return address is critical. 2. I also wanted to hand the parameters to the extension DLL processing function without having to copy massive amounts of data on each function call. 3. Because almost all the hook functions behave the same way, I set up the common variables needed in all functions. 4. Hook functions can't change any of the return values, including the value from GetLastError. By doing my own prolog and epilog, I can make it much easier to return the correct value. Also, I need to restore the register values to the state they were in following the real function call. A basic hook function looks like this: HANDLE NAKEDDEF DD_OpenEventA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) { // Any local variables for the function must be specified before // any other code. // HOOKFN_PROLOG must be specified right after the local variables. HOOKFN_PROLOG ( ) ; // Is the function type logging turned on? if ( TRUE == DoLogging ( DDOPT_EVENT ) ) { // Use the FILL_EVENTINFO macro to fill the stEvtInfo variable // that's declared in the HOOKFN_PROLOG macro. All hook // functions automatically have certain local variables to help // standardize their code. FILL_EVENTINFO ( eOpenEventA ) ; // You *MUST* call the REAL_FUNC_PRE_CALL macro BEFORE calling // the real function or else ESI and EDI won't be saved across // invocations. REAL_FUNC_PRE_CALL ( ) ; // Call the real function. The return value, stored in EAX, is // stored as part of the REAL_FUNC_POST_CALL processing. OpenEventA ( dwDesiredAccess , bInheritHandle , lpName ) ; // You *MUST* call the REAL_FUNC_POST_CALL macro AFTER calling // the real function. The register values and the last error // value are stored as part of REAL_FUNC_POST_CALL. REAL_FUNC_POST_CALL ( ) ; // Call the logging code to log the event. ProcessEvent ( &stEvtInfo ) ; } else { // See the comments above. The else clause handles the case // where the function isn't being logged. REAL_FUNC_PRE_CALL ( ) ; OpenEventA ( dwDesiredAccess , bInheritHandle , lpName ) ; REAL_FUNC_POST_CALL ( ) ; } // The last macro in the function is HOOKFN_EPILOG. The parameter // is the number of function parameters, so the stack will get // cleaned up correctly. The HOOKFN_EPILOG macro also takes care of // setting all the registers to the same values that the real // function returned. HOOKFN_EPILOG ( 3 ) ; } //////////////////////////////////////////////////////////////////////*/ /*////////////////////////////////////////////////////////////////////// The register state structure. I use this structure to ensure that ALL registers are returned exactly as they came from the real function. Notice that EBP and ESP are handled as part of the prolog. //////////////////////////////////////////////////////////////////////*/ typedef struct tag_REGSTATE { DWORD dwEAX ; DWORD dwEBX ; DWORD dwECX ; DWORD dwEDX ; DWORD dwEDI ; DWORD dwESI ; DWORD dwEFL ; } REGSTATE , * PREGSTATE ; /*////////////////////////////////////////////////////////////////////// The common prolog code for all DD_* functions //////////////////////////////////////////////////////////////////////*/ #define HOOKFN_PROLOG() \ /* All hook functions automatically get the same three local */\ /* variables. */\ DDEVENTINFO stEvtInfo ; /* The event information for the function*/\ DWORD dwLastError ; /* The last error value */\ REGSTATE stRegState ; /* The register state to ensure that I */\ /* restore the registers correctly */\ { \ __asm PUSH EBP /* Always save EBP explicitly. */\ __asm MOV EBP , ESP /* Move the stack. */\ __asm MOV EAX , ESP /* Get the stack pointer to calculate the*/\ /* return address and the parameters. */\ __asm SUB ESP , __LOCAL_SIZE /* Save space for the local variables.*/\ __asm ADD EAX , 04h + 04h /* Account for PUSH EBP and the */\ /* return address. */\ /* Save start of parameters on the stack.*/\ __asm MOV [stEvtInfo.dwParams] , EAX \ __asm SUB EAX , 04h /* Get back to the return address. */\ __asm MOV EAX , [EAX] /* EAX now holds the return address. */\ /* Save the return address. */\ __asm MOV [stEvtInfo.dwAddr] , EAX \ __asm MOV dwLastError , 0 /* Initialize dwLastError. */\ /* Initialize the event information. */\ __asm MOV [stEvtInfo.eFunc] , eUNINITIALIZEDFE \ __asm MOV [stRegState.dwEDI] , EDI /* Save the two registers that */\ __asm MOV [stRegState.dwESI] , ESI /* need to be saved across */\ /* function calls. */\ } /*////////////////////////////////////////////////////////////////////// The common epilog code for all DD_* functions. iNumParams is the number of parameters to the function that is used to restore the stack to the proper place after the hook call. //////////////////////////////////////////////////////////////////////*/ #define HOOKFN_EPILOG(iNumParams) \ { \ SetLastError ( dwLastError ) ; /* Set the real function's last */\ /* error value. */\ __asm ADD ESP , __LOCAL_SIZE /* Add back the local variables. */\ __asm MOV EBX , [stRegState.dwEBX]/* Restore all the registers so */\ __asm MOV ECX , [stRegState.dwECX]/* that this call looks */\ __asm MOV EDX , [stRegState.dwEDX]/* identical to the intercepted */\ __asm MOV EDI , [stRegState.dwEDI]/* function. */\ __asm MOV ESI , [stRegState.dwESI] \ __asm MOV EAX , [stRegState.dwEFL] \ __asm SAHF \ __asm MOV EAX , [stRegState.dwEAX] \ __asm MOV ESP , EBP /* Put back ESP. */\ __asm POP EBP /* Restore the saved EBP. */\ __asm RET iNumParams * 4 /* stdcall restore of the stack */\ } /*////////////////////////////////////////////////////////////////////// The REAL_FUNC_PRE_CALL macro needs to be placed IMMEDIATELY *BEFORE* ANY call to the real function the hook function is handling. The macro ensures that EDI and ESI are returned in the same condition they were in when passed to the hook function. //////////////////////////////////////////////////////////////////////*/ #define REAL_FUNC_PRE_CALL() \ { \ __asm MOV EDI , [stRegState.dwEDI] /* Restore the real EDI. */\ __asm MOV ESI , [stRegState.dwESI] /* Restore the real ESI. */\ } /*////////////////////////////////////////////////////////////////////// The REAL_FUNC_POST_CALL macro needs to be placed IMMEDIATELY *AFTER* ANY call to the real function the hook function is handling. All the register values after the real call are saved so that the hook function epilog can return the same register values as the real function call. //////////////////////////////////////////////////////////////////////*/ #define REAL_FUNC_POST_CALL() \ { \ __asm MOV [stRegState.dwEAX] , EAX /* Save the EAX value. */\ __asm MOV [stRegState.dwEBX] , EBX /* Save the EBX value. */\ __asm MOV [stRegState.dwECX] , ECX /* Save the ECX value. */\ __asm MOV [stRegState.dwEDX] , EDX /* Save the EDX value. */\ __asm MOV [stRegState.dwEDI] , EDI /* Save the EDI value. */\ __asm MOV [stRegState.dwESI] , ESI /* Save the ESI value. */\ __asm XOR EAX , EAX /* Zero out EAX. */\ __asm LAHF /* Load the flag values into AH.*/\ __asm MOV [stRegState.dwEFL] , EAX /* Save the flag values. */\ } \ dwLastError = GetLastError ( ) ; /* Save the last error value. */\ { \ __asm MOV EAX , [stRegState.dwEAX] /* Restore EAX to its original */\ /* value. */\ /* Set the return value for */\ /* the information. */\ __asm MOV [stEvtInfo.dwRetValue] , EAX \ } /*////////////////////////////////////////////////////////////////////// A convenient macro to fill out the event information structure //////////////////////////////////////////////////////////////////////*/ #define FILL_EVENTINFO(eFn) \ stEvtInfo.eFunc = eFn ; \ stEvtInfo.ePrePost = ePostCall ; \ stEvtInfo.dwThreadId = GetCurrentThreadId ( ) /*////////////////////////////////////////////////////////////////////// The declaration for all DD_* definitions //////////////////////////////////////////////////////////////////////*/ #define NAKEDDEF __declspec(naked) /*////////////////////////////////////////////////////////////////////// BIG NOTE BIG NOTE BIG NOTE BIG NOTE All the following prototypes look like cdecl functions. They are not--they are all stdcall! The custom prolog and epilog ensure that the correct calling convention is used! //////////////////////////////////////////////////////////////////////*/ //////////////////////////////////////////////////////////////////////// // The mandatory functions that have to be intercepted to make the // system work HMODULE DD_LoadLibraryA ( LPCSTR lpLibFileName ) ; HMODULE DD_LoadLibraryW ( LPCWSTR lpLibFileName ) ; HMODULE DD_LoadLibraryExA ( LPCSTR lpLibFileName , HANDLE hFile , DWORD dwFlags ) ; HMODULE DD_LoadLibraryExW ( LPCWSTR lpLibFileName , HANDLE hFile , DWORD dwFlags ) ; VOID DD_ExitPr FARPROC DD_GetProcAddress ( HMODULE hModule , LPCSTR lpProcName ) ; //////////////////////////////////////////////////////////////////////// // The thread-specific functions HANDLE DD_CreateThread (LPSECURITY_ATTRIBUTES lpThreadAttributes , DWORD dwStackSize , LPTHREAD_START_ROUTINE lpStartAddress , LPVOID lpParameter , DWORD dwCreationFlags , LPDWORD lpThreadId ) ; VOID DD_ExitThread ( DWORD dwExitCode ) ; DWORD DD_SuspendThread ( HANDLE hThread ) ; DWORD DD_ResumeThread ( HANDLE hThread ) ; BOOL DD_TerminateThread ( HANDLE hThread , DWORD dwExitCode ) ; //////////////////////////////////////////////////////////////////////// // Waiting and special functions DWORD DD_WaitForSingleObject ( HANDLE hHandle , DWORD dwMilliseconds ) ; DWORD DD_WaitForSingleObjectEx ( HANDLE hHandle , DWORD dwMilliseconds , BOOL bAlertable ) ; DWORD DD_WaitForMultipleObjects( DWORD nCount , CONST HANDLE * lpHandles , BOOL bWaitAll , DWORD dwMilliseconds ) ; DWORD DD_WaitForMultipleObjectsEx( DWORD nCount , CONST HANDLE * lpHandles , BOOL bWaitAll , DWORD dwMilliseconds , BOOL bAlertable ) ; DWORD DD_MsgWaitForMultipleObjects ( DWORD nCount , LPHANDLE pHandles , BOOL fWaitAll , DWORD dwMilliseconds, DWORD dwWakeMask ) ; DWORD DD_MsgWaitForMultipleObjectsEx ( DWORD nCount , LPHANDLE pHandles , DWORD dwMilliseconds , DWORD dwWakeMask , DWORD dwFlags ) ; DWORD DD_SignalObjectAndWait ( HANDLE hObjectToSignal , HANDLE hObjectToWaitOn , DWORD dwMilliseconds , BOOL bAlertable ) ; BOOL DD_CloseHandle ( HANDLE hObject ) ; //////////////////////////////////////////////////////////////////////// // Critical-section functions VOID DD_InitializeCriticalSection(LPCRITICAL_SECTION lpCriticalSection); BOOL DD_InitializeCriticalSectionAndSpinCount ( LPCRITICAL_SECTION lpCriticalSection, DWORD dwSpinCount ); VOID DD_DeleteCriticalSection(LPCRITICAL_SECTION lpCriticalSection ) ; VOID DD_EnterCriticalSection ( LPCRITICAL_SECTION lpCriticalSection ) ; VOID DD_LeaveCriticalSection ( LPCRITICAL_SECTION lpCriticalSection ) ; DWORD DD_SetCriticalSectionSpinCount ( LPCRITICAL_SECTION lpCriticalSection, DWORD dwSpinCount ); BOOL DD_TryEnterCriticalSection ( LPCRITICAL_SECTION lpCriticalSection); //////////////////////////////////////////////////////////////////////// // Mutex functions HANDLE DD_CreateMutexA ( LPSECURITY_ATTRIBUTES lpMutexAttributes , BOOL bInitialOwner , LPCSTR lpName ) ; HANDLE DD_CreateMutexW ( LPSECURITY_ATTRIBUTES lpMutexAttributes , BOOL bInitialOwner , LPCWSTR lpName ) ; HANDLE DD_OpenMutexA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenMutexW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_ReleaseMutex ( HANDLE hMutex ) ; //////////////////////////////////////////////////////////////////////// // Semaphore functions HANDLE DD_CreateSemaphoreA ( LPSECURITY_ATTRIBUTES lpSemaphoreAttributes , LONG lInitialCount , LONG lMaximumCount , LPCSTR lpName ); HANDLE DD_CreateSemaphoreW ( LPSECURITY_ATTRIBUTES lpSemaphoreAttributes , LONG lInitialCount , LONG lMaximumCount , LPCWSTR lpName ); HANDLE DD_OpenSemaphoreA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenSemaphoreW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_ReleaseSemaphore ( HANDLE hSemaphore , LONG lReleaseCount , LPLONG lpPreviousCount ) ; //////////////////////////////////////////////////////////////////////// // Event functions HANDLE DD_CreateEventA ( LPSECURITY_ATTRIBUTES lpEventAttributes , BOOL bManualReset , BOOL bInitialState , LPCSTR lpName ) ; HANDLE DD_CreateEventW ( LPSECURITY_ATTRIBUTES lpEventAttributes , BOOL bManualReset , BOOL bInitialState , LPCWSTR lpName ) ; HANDLE DD_OpenEventA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenEventW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_PulseEvent ( HANDLE hEvent ) ; BOOL DD_ResetEvent ( HANDLE hEvent ) ; BOOL DD_SetEvent ( HANDLE hEvent ) ; #endif // _DD_FUNCS_H |
I want to mention several other minor points about DeadlockDetection. The first is that DeadlockDetection is always active in your application, even if you suspend DeadlockDetection logging. Instead of hooking and unhooking dynamically, I leave the functions hooked and look at some internal flags to determine how the hook should behave. Keeping all functions hooked makes it easier to toggle different function logging at run time, but it adds some overhead to your application. I felt that allowing hooking and unhooking on the fly would have led to more errors in the DeadlockDetection code.
Second, DeadlockDetection hooks the functions out of a DLL when brought into your program through LoadLibrary. However, it can gain control only after that DLL's DllMain has executed, so if any synchronization objects are created or used during DllMain, DeadlockDetection can miss them.
Third, DeadlockDetection also hooks GetProcAddress and ExitProcess. The GetProcAddress hooking is there in case your program, or a third-party control you might be deadlocking against, calls GetProcAddress to find a synchronization method at run time.
I hook ExitProcess because when the application is ending, I need to unhook and shut down DeadlockDetection so that it doesn't crash or hang your program. Because there's no way to control the unload order of DLLs during program termination, you can easily get into situations in which a DLL that DeadlockDetection relies on, such as DeadDetExt, has been unloaded before DeadlockDetection itself. Fortunately, very few developers are doing major multithreaded control after the application calls ExitProcess.
Finally, several test programs are included with DeadlockDetection on the companion CD. All of them are included in the main DeadlockDetection workspace, and all link against DEADLOCKDETECTION.DLL, so you can use them to see how DeadlockDetection operates.
EAN: 2147483647
Pages: 122