Implementing DeadlockDetection
|
As you can see, using DeadlockDetection is fairly simple. Beneath the simplicity of use, however, is a fairly sophisticated implementation. The first part of DeadlockDetection's implementation I'll go over is how to hook functions.
Hooking Imported Functions
There are many ways to hook the function calls a program makes. The hard way is to hunt down all the CALL instructions and replace the address with one of your own. This approach is difficult and extremely error prone. Fortunately, with DeadlockDetection, the functions I need to hook are imported functions and are much easier to handle than CALL instructions.
An imported function is a function that comes out of a DLL. For example, when your program calls OutputDebugString, it's calling a function that resides in KERNEL32.DLL. When I first started doing Win32-based programming, I thought that calling an imported function would be just like calling any other function—a CALL or branch instruction would jump to an address and start executing the imported function. The only difference might be that with an imported function, the operating system program loader would have to run through the executable and fix up the addresses to reflect where the DLL being called would be loaded into memory. When I looked at how a call to an imported function really is implemented, I was amazed at the simplicity and beauty of the design.
The problem with the way I was thinking becomes apparent when you consider how many API functions there are and that you can easily call the same ones many times throughout your program. If the loader had to find and replace each occurrence of a call to OutputDebugString, for example, loading a program could take forever. Even if the linker generated a complete table that specified where each call to OutputDebugString took place in the code, the huge amount of looping and memory writing would make load times excruciatingly slow.
So how does the loader tell your application where to find an imported function? The solution is fiendishly clever. If you think about where the calls to OutputDebugString go, you'll soon realize that each call must go to the same address: the address where OutputDebugString is loaded into memory. Of course, your application can't know this address ahead of time, so instead, all your OutputDebugString calls get routed through a single, indirect address. When the program loader loads your executable and its dependent DLLs, the loader fixes up this one indirect address so that it corresponds to the final load address of OutputDebugString. The compiler makes this indirect addressing work by generating a jump to the indirect address any time your code calls the imported function. This indirect address is stored in the .idata (or import) section of the executable. If you import through __declspec(dllimport), instead of being an indirect jump, the code is an indirect call, thus saving a couple of instructions per function call.
Hooking an imported function is a matter of finding the imports section of the executable, looking for the address of the particular function you want to hook, and then writing the hook function address in its place. Although hunting down and replacing function addresses might sound like a lot of work, it's not that bad because the Win32 Portable Executable (PE) file format is so nicely organized.
In Chapter 10 of his excellent book Windows 95 System Programming Secrets (IDG Books, 1995), Matt Pietrek describes a method for hooking imported functions. Matt's code simply finds a module's imports section and, using the value returned from a call to GetProcAddress, loops through the imported functions. When Matt's code finds the function it's looking for, it overwrites the original imported function address with the hook function.
Not surprisingly, given that 1995 is a past lifetime in software development circles, two small issues have changed since Matt's book came out. The first is that when Matt wrote his book, most developers didn't merge their imports section with other PE sections. Therefore, if the imports section is in read-only memory, you cause an access violation by writing the hook address. I fixed the read-only memory issue by setting the virtual memory protection to read-write before the hook function address is written to memory. The second issue, which is a little tougher to work around, is that under certain conditions you can't hook imported functions under Microsoft Windows Me. Since so many people ask me about how to hook functions, I decided to keep the Microsoft Windows Me support for hooking code, so it's relevant for me to discuss what happens under Windows Me.
When you use DeadlockDetection, you want to be able to have the threading functions redirected any time you run your application, even when the application is running under the debugger. Although you wouldn't think that hooking functions while running under a debugger would be a problem, it is. In Windows XP, or when running a program in Windows Me outside a debugger, when you call GetProcAddress to find a function and then look through the imports section for that address, you'll always find it. But under Windows Me, calling GetProcAddress in your program while it's running under a debugger returns a different address than when it runs outside a debugger. What GetProcAddress actually returns when running under the debugger is a debug thunk—a special wrapper around the real call.
The reason for the debug thunk is that Windows Me doesn't implement copy-on-write in the operating system for addresses above 2 GB. Copy-on-write means that when a page of shared memory is written to, the operating system makes a copy of that page and gives it to the process where the write is taking place. For the most part, Windows Me and Windows XP follow the same rules and everything is fine. However, for shared memory above 2 GB, where all the operating system DLLs load on Windows Me, Windows Me doesn't do the copy-on-write for those addresses. That means that when you change the memory inside an operating system DLL to set a breakpoint or to patch a function, it's patched for all processes in the system and will cause a complete system crash if another process executes at that changed location. Therefore, Windows Me goes to quite a bit of trouble to keep you from accidentally messing up that memory.
The debug thunk returned by GetProcAddress when running under a debugger is a means by which Windows Me keeps debuggers from attempting to step into system functions above the 2-GB line. Overall, the lack of copy-on-write isn't much of an issue for most developers—only those who write debuggers or who want to hook functions correctly whether or not they're running under a debugger.
Fortunately, getting the real address for an imported function isn't too difficult—it just takes a little more work, and you have to avoid GetProcAddress. The PE file IMAGE_IMPORT_DESCRIPTOR structure, which holds all the information about functions imported from a specific DLL, has pointers to two arrays in the executable. These arrays are called import address tables (IATs), or sometimes thunk data arrays. The first pointer references the real IAT, which the program loader fixes up when the executable is loaded. The second pointer references the original IAT, which is untouched by the loader and lists the imported functions. To find the real imported function address, simply work your way through the original IAT until you find the named function that you want to hook, and then write the hook address in the corresponding entry in the real IAT, which the program is using. By taking this extra step, the hooking code will always work no matter where it's called.
HookImportedFunctionsByName is the function I wrote to take care of your hooking needs. Table 15-3 shows the parameters to HookImportedFunctionsByName and describes each one. Because I wanted to make the hooking as generic as possible, I went to the trouble of allowing you to hook multiple functions imported from the same DLL at the same time. As its name implies, the HookImportedFunctionsByName function will hook only those functions imported by name. Though I won't be discussing it in this book, I did include another function, HookOrdinalExport, that allows you to hook functions exported by ordinal.
| Parameter | Description |
|---|---|
| hModule | The module in which the imports will be hooked. |
| szImportMod | The name of the module whose functions are imported. |
| UiCount | The number of functions to hook. This parameter is the size of the paHookArray and paOrigFuncs arrays. |
| paHookArray | The array of function descriptor structures that indicates which functions to hook. The array doesn't have to be in szFunc name order (though it's wise to keep the array sorted in function name order, because I might implement better searching in the future). Also, if a particular pProc is NULL, HookImportedFunctionsByName skips that item. The structure for each element in paHookArray is simply the name of the function to hook and a pointer to the new hook procedure. Because you might want to hook or unhook functions at will, HookImportedFunctionsByName returns all the original imported function addresses. |
| paOrigFuncs | The array of original addresses hooked by HookImportedFunctionsByName. If a function wasn't hooked, that item index will be NULL. |
| pdwHooked | Returns the number of functions hooked out of paHookArray. |
Listing 15-2 shows the HookImportedFunctionsByNameA function. In your code, you'll call HookImportedFunctionsByName because that macro maps between ANSI and Unicode. However, since the names in the IAT section are all stored as ANSI, the implementation for HookImportedFunctionsByNameW simply converts the appropriate parameters to ANSI and calls HookImportedFunctionsByNameA.
Listing 15-2: HookImportedFunctionsByNameA from HOOKIMPORTEDFUNCTIONBYNAME.CPP
 |
BOOL BUGSUTIL_DLLINTERFACE __stdcall HookImportedFunctionsByNameA ( HMODULE hModule , LPCSTR szImportMod , UINT uiCount , LPHOOKFUNCDESC paHookArray , PROC * paOrigFuncs , LPDWORD pdwHooked ) { // Assert the parameters. ASSERT ( FALSE == IsBadReadPtr ( hModule , sizeof ( IMAGE_DOS_HEADER ) ) ) ; ASSERT ( FALSE == IsBadStringPtrA ( szImportMod , MAX_PATH ) ) ; ASSERT ( 0 != uiCount ) ; ASSERT ( NULL != paHookArray ) ; ASSERT ( FALSE == IsBadReadPtr ( paHookArray , sizeof (HOOKFUNCDESC) * uiCount )); // In debug builds, perform deep validation of paHookArray. #ifdef _DEBUG if ( NULL != paOrigFuncs ) { ASSERT ( FALSE == IsBadWritePtr ( paOrigFuncs , sizeof ( PROC ) * uiCount ) ); } if ( NULL != pdwHooked ) { ASSERT ( FALSE == IsBadWritePtr ( pdwHooked , sizeof ( UINT ))); } // Check each item in the hook array. { for ( UINT i = 0 ; i < uiCount ; i++ ) { ASSERT ( NULL != paHookArray[ i ].szFunc ) ; ASSERT ( '\0' != *paHookArray[ i ].szFunc ) ; // If the function address isn't NULL, it is validated. if ( NULL != paHookArray[ i ].pProc ) { ASSERT ( FALSE == IsBadCodePtr ( paHookArray[i].pProc)); } } } #endif // Perform the error checking for the parameters. if ( ( 0 == uiCount ) || ( NULL == szImportMod ) || ( TRUE == IsBadReadPtr ( paHookArray , sizeof ( HOOKFUNCDESC ) * uiCount ) )) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } if ( ( NULL != paOrigFuncs ) && ( TRUE == IsBadWritePtr ( paOrigFuncs , sizeof ( PROC ) * uiCount ) ) ) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } if ( ( NULL != pdwHooked ) && ( TRUE == IsBadWritePtr ( pdwHooked , sizeof ( UINT ) ) ) ) { SetLastErrorEx ( ERROR_INVALID_PARAMETER , SLE_ERROR ) ; return ( FALSE ) ; } // Is this a system DLL above the 2-GB line, which Windows 98 won't // let you patch? if ( ( FALSE == IsNT ( ) ) && ( (DWORD_PTR)hModule >= 0x80000000 ) ) { SetLastErrorEx ( ERROR_INVALID_HANDLE , SLE_ERROR ) ; return ( FALSE ) ; } // TODO TODO // Should each item in the hook array be checked in release builds? if ( NULL != paOrigFuncs ) { // Set all the values in paOrigFuncs to NULL. memset ( paOrigFuncs , NULL , sizeof ( PROC ) * uiCount ) ; } if ( NULL != pdwHooked ) { // Set the number of functions hooked to 0. *pdwHooked = 0 ; } // Get the specific import descriptor. PIMAGE_IMPORT_DESCRIPTOR pImportDesc = GetNamedImportDescriptor ( hModule , szImportMod ); if ( NULL == pImportDesc ) { // The requested module wasn't imported. Don't return an error. return ( TRUE ) ; } // BUG FIX (thanks to Attila Szepesv ry!) // Make sure the original first thunk and first thunk are not // NULL. The original first thunk can be a null import // descriptor and cause this routine to crash. if ( ( NULL == pImportDesc->OriginalFirstThunk ) || ( NULL == pImportDesc->FirstThunk ) ) { // I return TRUE here because it's analogous to the case in // which the requested module wasn't imported. // All OK, JumpMaster! SetLastError ( ERROR_SUCCESS ) ; return ( TRUE ) ; } // Get the original thunk information for this DLL. I can't use // the thunk information stored in pImportDesc->FirstThunk // because the loader has already changed that array to fix up // all the imports. The original thunk gives me access to the // function names. PIMAGE_THUNK_DATA pOrigThunk = MakePtr ( PIMAGE_THUNK_DATA , hModule , pImportDesc->OriginalFirstThunk ) ; // Get the array the pImportDesc->FirstThunk points to because // I'll do the actual bashing and hooking there. PIMAGE_THUNK_DATA pRealThunk = MakePtr ( PIMAGE_THUNK_DATA , hModule , pImportDesc->FirstThunk ); // Loop through and find the functions to hook. while ( NULL != pOrigThunk->u1.Function ) { // Look only at functions that are imported by name, not those // that are imported by ordinal value. if ( IMAGE_ORDINAL_FLAG != ( pOrigThunk->u1.Ordinal & IMAGE_ORDINAL_FLAG )) { // Look at the name of this imported function. PIMAGE_IMPORT_BY_NAME pByName ; pByName = MakePtr ( PIMAGE_IMPORT_BY_NAME , hModule , pOrigThunk->u1.AddressOfData ) ; // If the name starts with NULL, skip it. if ( '\0' == pByName->Name[ 0 ] ) { // BUG FIX (thanks to Attila Szepesv ry!) // I forgot to increment the thunk pointers! pOrigThunk++ ; pRealThunk++ ; continue ; } // Determines whether I hook the function BOOL bDoHook = FALSE ; // TODO TODO // Might want to consider bsearch here. // See whether the imported function name is in the hook // array. Consider requiring paHookArray to be sorted by // function name so that bsearch can be used, which // will make the lookup faster. However, the size of // uiCount coming into this function should be rather // small, so it's OK to search the entire paHookArray for // each function imported by szImportMod. for ( UINT i = 0 ; i < uiCount ; i++ ) { if ( ( paHookArray[i].szFunc[0] == pByName->Name[0] ) && ( 0 == strcmpi ( paHookArray[i].szFunc , (char*)pByName->Name ) ) ) { // If the function address is NULL, exit now; // otherwise, go ahead and hook the function. if ( NULL != paHookArray[ i ].pProc ) { bDoHook = TRUE ; } break ; } } if ( TRUE == bDoHook ) { // I found a function to hook. Now I need to change // the memory protection to writable before I overwrite // the function pointer. Note that I'm now writing into // the real thunk area! MEMORY_BASIC_INFORMATION mbi_thunk ; VirtualQuery ( pRealThunk , &mbi_thunk , sizeof ( MEMORY_BASIC_INFORMATION ) ) ; if ( FALSE == VirtualProtect ( mbi_thunk.BaseAddress , mbi_thunk.RegionSize , PAGE_READWRITE , &mbi_thunk.Protect )) { ASSERT ( !"VirtualProtect failed!" ) ; SetLastErrorEx ( ERROR_INVALID_HANDLE , SLE_ERROR ); return ( FALSE ) ; } // Save the original address if requested. if ( NULL != paOrigFuncs ) { paOrigFuncs[i] = (PROC)((INT_PTR)pRealThunk->u1.Function) ; } // Hook the function. DWORD_PTR * pTemp = (DWORD_PTR*)&pRealThunk->u1.Function ; *pTemp = (DWORD_PTR)(paHookArray[i].pProc); DWORD dwOldProtect ; // Change the protection back to what it was before I // overwrote the function pointer. VERIFY ( VirtualProtect ( mbi_thunk.BaseAddress , mbi_thunk.RegionSize , mbi_thunk.Protect , &dwOldProtect ) ) ; if ( NULL != pdwHooked ) { // Increment the total number of functions hooked. *pdwHooked += 1 ; } } } // Increment both tables. pOrigThunk++ ; pRealThunk++ ; } // All OK, JumpMaster! SetLastError ( ERROR_SUCCESS ) ; return ( TRUE ) ; } |
HookImportedFunctionsByName shouldn't be difficult for you to follow. After doing the proactive debugging practice of validating every parameter thoroughly, I call the helper function GetNamedImportDescriptor to find the IMAGE_IMPORT_DESCRIPTOR for the requested module. After getting the pointers to the original and real IATs, I loop through the original IAT, looking at each function imported by name to see whether it's in the paHookArray list. If the function is in the hook list, I simply set the real IAT's memory to PAGE_READWRITE so that the hooked address can be written safely, write the hook into the entry for the real function, and reset the memory protection to its original setting. The unit test function for HookImportedFunctionsByName is included with the 
BUGSLAYERUTIL.DLL source code, so feel free to use it to step through the code if you don't quite follow what's going on.
Now that you have an idea of how to hook the imported functions, let's move on to implementing the rest of DeadlockDetection.
Implementation Highlights
One of my primary goals in implementing DeadlockDetection was to make the utility as data-driven and table-driven as possible. When you step back and look at what the DLL does in its hook processing, you'll see that the processing is almost identical for each function in Table 15-1. The hooked function gets called, determines whether its class of functions is being monitored, calls the real function, and (if logging is on for that class) logs the information and returns. I had to write a bunch of similar hook functions, and I wanted to make them as simple as possible. Complicated hook functions are a perfect breeding ground for bugs, and even when you try to make things simple, bugs can creep in. In a moment, I'll point out where I had a nasty bug in the DeadlockDetection code for the first edition of this book.
The best way to show this simplicity is to talk about writing a DeadDetExt DLL. A DeadDetExt DLL must have three exported functions. The first two, DeadDetExtOpen and DeadDetExtClose, are self-explanatory. The interesting function is DeadDetProcessEvent, which each hook function calls when there is information to write. DeadDetProcessEvent takes a single parameter, a pointer to a DDEVENTINFO structure:
typedef struct tagDDEVENTINFO { // The identifier that specifies what is contained in the rest of // this structure. eFuncEnum eFunc ; // The pre-call or post-call indicator. ePrePostEnum ePrePost ; // The return address. This is so the calling function can be // found. DWORD dwAddr ; // The thread ID of the calling thread. DWORD dwThreadId ; // The return value for post calls DWORD dwRetValue ; // The parameter information. Cast this to the appropriate // structure as, described below, for the function. When accessing // the parameters, treat them as read-only. DWORD dwParams ; } DDEVENTINFO , * LPDDEVENTINFO ; The entire output for a single function that appears in Listing 15-1 comes from the information in the DDEVENTINFO structure. Although most of the fields in DDEVENTINFO are self-explanatory, the dwParams field needs special mention. The dwParams field is really a pointer to the parameters as they appear in memory.
In Chapter 7, I discussed how parameters are passed on the stack. Just to jog your memory, parameters for __stdcall and __cdecl functions are passed right to left and the stack grows from high memory to low memory. The dwParams field in the DDEVENTINFO structure points to the last parameter on the stack, so the structure lists the parameters in left-to-right order. I applied a little creative casting to make it easy to convert dwParams.
In DEADLOCKDETECTION.H, I provide typedefs that describe each intercepted function's parameter lists. For example, if eFunc were eWaitForSingleObjectEx, you would cast dwParams to LPWAITFORSINGLEOBJECTEX_PARAMS to get the parameters. To see all this creative casting in action, check out the TEXTFILEDDEXT.DLL code included with the sample files.
Although output processing is relatively easy, gathering the information can be difficult. I wanted DeadlockDetection to hook the synchronization functions in Table 15-1, but I didn't want the hook functions to change the behavior of the real functions. I also wanted to get the parameters and the return value and to write the hook functions in C/C++ easily. I spent quite a while with the debugger and the disassembler before I got it right.
Initially, I wrote all the hook functions so that they were just pass-through functions and called the real functions directly. This approach worked great. Then I put the parameters and the return value for the functions into local variables. Getting the value returned from the real function was simple, but because I started implementation of DeadlockDetection with Visual C++ 6, I didn't have a clean way to get the return address in my C/C++ hook function. Visual C++ .NET now offers the _ReturnAddress intrinsic, but that was not available in Visual C++ 6. I needed the DWORD right before the current stack pointer. Unfortunately, in straight C/C++, the function prolog would've already done its magic by the time I could get control, so the stack pointer would've already moved away from where it needed to be.
You might think that the stack pointer is just offset by the number of local variables, but that isn't always the case. The Visual C++ compiler does an excellent job of optimizing so that the stack pointer isn't in the same place with different optimization flags set. Although you might declare a variable as a local variable, the compiler can optimize the variable by storing it in a register so that it doesn't even appear on the stack.
I needed a guaranteed way to get the stack pointer no matter what optimizations were set. At this point, I started thinking naked (no, not me without clothes): why not declare the hook functions as __declspec(naked) and create my own prolog and epilog code? With this approach, I'd have complete control over ESP no matter what optimization settings were used. Additionally, getting the return address and parameters would be a snap because they are at ESP+04h and ESP+08h, respectively. Keep in mind that I'm not doing anything out of the ordinary with the prolog and epilog code, so I still perform the usual PUSH EBP and MOV EBP, ESP for the prolog and MOV ESP, EBP and POP EBP for the epilog.
Because each hook function was going to be declared as __declspec(naked), I wrote a couple of macros to handle the prolog and epilog: HOOKFN_PROLOG and HOOKFN_EPILOG. I also went ahead and declared some common local variables that all hook functions would need in HOOKFN_PROLOG. These variables included the last error value, dwLastError, and the event information structure to pass to the DeadDetExt DLL, stEvtInfo. The dwLastError is yet another bit of state that I needed to preserve when intercepting functions.
The Windows API can return a special error code through SetLastError to provide more information if a function fails. This error code can be a real boon because it tells you why an API function failed. For example, if GetLastError returns 122, you know that the buffer parameter was too small. WINERROR.H contains all the error codes the operating system returns. The problem with hook functions is that they can reset the last error as part of their processing. This behavior can wreak havoc if your application relies on the last error.
If you call CreateEvent and want to see whether the returned handle was created or just opened, CreateEvent sets the last error to ERROR_ALREADY_EXISTS if it just opened the handle. Because the cardinal rule of intercepting functions is that you can't change the expected behavior of the function, I needed to call GetLastError immediately after the real function call so that my hook function could properly set the last error code that the real function returned. The general rule for a hook function is that you need to call GetLastError right after you call the real function and then call SetLastError as the last action before leaving the hook function.
|
When it came time to update DeadlockDetection for the second edition of this book, I thought I might want to go through and update the HOOKFN_PROLOG macro to use the new _ReturnAddress intrinsic to make my life a little easier. That would mean I could get rid of the naked declarations and make the functions more normal, and not rely on the special prolog and epilog. However, the one great advantage I get with the existing macros is that I can treat the parameters as a blob in memory and pass them directly to the output function. If I were to use standard functions, I'd have to require some sort of weird address of casting to achieve the same thing. I also had the most powerful argument of all: the code was working quite well and I wasn't going to buy sufficient advantage to consider rewriting the code. Consequently, I left the naked functions as they were.
|
At this point, I thought I was done except for testing. Unfortunately, my first test uncovered a bug: I wasn't preserving ESI and EDI across the hook call because the documentation on using the inline assembler explicitly stated that you didn't have to save them. After I fixed the ESI/EDI register problem, DeadlockDetection seemed to work fine. When I started doing register comparisons on before, during, and after cases, however, I noticed that I wasn't returning the values the real functions left in EBX, ECX, and EDX, and worse yet, in the flags' registers. Although I didn't see any problems and the documentation said that those registers didn't need to be preserved, I still was concerned that my hook functions were changing the application state. I declared the REGSTATE structure to hold the register values after the real function call so that I could restore the values when my hook function returned. To save and restore the registers, I created two additional macros, REAL_FUNC_PRE_CALL and REAL_FUNC_POST_CALL, which I placed around the real call the hook function makes.
After a little more testing, I found another problem: in release builds with full optimizations, I crashed inexplicably every so often. I finally tracked down those crashes and was able to attribute them to the effect of the optimizer on some of my hook functions. The optimizer was trying to be helpful but ended up doing more harm than good. I was very careful about the register usage in my hook functions and used only EAX or stack memory directly. Even though I was taking every precaution to preserve the registers, I found that the debug build code sequence
MOV DWORD PTR [EBP-018h] , 00000002h MOV DWORD PTR [EBP-014h] , 00000002h
was being transformed by the optimizer into
PUSH 002h POP EBX MOV DWORD PTR [EBP-01Ch] , EBX MOV DWORD PTR [EBP-018h] , EBX
It's easy to see in the second snippet that the POP into EBX was trashing the register. To prevent the optimizer from stealing registers behind my back, I turned optimizations off for all hook function files by placing
#pragma optimize("", off ) at the top of each file. Turning optimizations off also made debugging easier because the unoptimized code the compiler generates is very similar for both release and debug builds.
Listing 15-3 shows the final version of DD_FUNCS.H, which is the internal header file in which all the special hook function macros are declared. The comment at the top of the file has two sample hook functions that explain how I used each of the special macros. I strongly encourage you to step through the DDSimpTest example that's part of the source code. Make sure that you watch an entire function call at the assembly-language level because that's the only way you'll see all the processing that takes place.
Listing 15-3: DD_FUNCS.H
|
/*---------------------------------------------------------------------- Debugging Applications for Microsoft .NET and Microsoft Windows Copyright 1997-2003 John Robbins -- All rights reserved. ------------------------------------------------------------------------ The prototypes for all the hook functions and the prolog/epilog code ----------------------------------------------------------------------*/ #ifndef _DD_FUNCS_H #define _DD_FUNCS_H /*////////////////////////////////////////////////////////////////////// All the hook functions are __declspec(naked) functions, so I must provide the prolog and epilog. I need to provide a custom prolog and epilog for several reasons: 1. Functions written in C have no control over which registers are used or when the compiler saves the original registers. Not having control over the registers means that getting the return address is nearly impossible. For the DeadlockDetection project, the return address is critical. 2. I also wanted to hand the parameters to the extension DLL processing function without having to copy massive amounts of data on each function call. 3. Because almost all the hook functions behave the same way, I set up the common variables needed in all functions. 4. Hook functions can't change any of the return values, including the value from GetLastError. By doing my own prolog and epilog, I can make it much easier to return the correct value. Also, I need to restore the register values to the state they were in following the real function call. A basic hook function only requires two macros, HOOKFN_STARTUP and HOOKFN_SHUTDOWN. As you can see, doing a regular function is quite easy! BOOL NAKEDDEF DD_InitializeCriticalSectionAndSpinCount ( LPCRITICAL_SECTION lpCriticalSection, DWORD dwSpinCount ) { HOOKFN_STARTUP ( eInitializeCriticalSectionAndSpinCount , DDOPT_CRITSEC , 0 ) ; InitializeCriticalSectionAndSpinCount ( lpCriticalSection , dwSpinCount ) ; HOOKFN_SHUTDOWN ( 2 , DDOPT_CRITSEC ) ; } If you have to do special processing and don't want to do something the normal macros don't do, you can use the following macros to do the necessary behind-the-scenes work: HOOKFN_PROLOG REAL_FUNC_PRE_CALL REAL_FUNC_POST_CALL HOOKFN_EPILOG Here's an example of a function using the macros: HMODULE NAKEDDEF DD_LoadLibraryA ( LPCSTR lpLibFileName ) { // Any local variables must be declared before the HOOKFN_PROLOG // macro. This sets up the actual function prolog and automatically // defines some key variables such as stEvtInfo (DDEVENTINFO). HOOKFN_PROLOG ( ) ; // Before doing the call to the real function, you need to specify // the REAL_FUNC_PRE_CALL macro to ensure the registers get set // up to reflect what they were when the function was called. REAL_FUNC_PRE_CALL ( ) ; // Make the real function call. LoadLibraryA ( lpLibFileName ) ; // The macro to save off the registers after the real call. This // way I can change the registers before I leave so that it appears // this function wasn't even there. REAL_FUNC_POST_CALL ( ) ; // The work specific to the LoadLibraryA hook. if ( NULL != stEvtInfo.dwRetValue ) { HookAllLoadedModules ( ) ; } // Takes care of the function epilog, restoring the registers as // well as the stack. The number passed to the macro is the // number of parameters passed to the function. Be very careful // and make sure you don't get "editor inheritance" errors when you // cut and paste this macro into other functions! HOOKFN_EPILOG ( 1 ) ; } //////////////////////////////////////////////////////////////////////*/ /*////////////////////////////////////////////////////////////////////// // The register state structure. I use this structure to ensure that // ALL registers are returned exactly as they came from the real // function. Notice that EBP and ESP are handled as part of the // prolog. //////////////////////////////////////////////////////////////////////*/ typedef struct tag_REGSTATE { DWORD dwEAX ; DWORD dwEBX ; DWORD dwECX ; DWORD dwEDX ; DWORD dwEDI ; DWORD dwESI ; DWORD dwEFL ; } REGSTATE , * PREGSTATE ; /*////////////////////////////////////////////////////////////////////// // The macros to save and restore ESI across function calls in debug // builds. //////////////////////////////////////////////////////////////////////*/ #ifdef _DEBUG #define SAVE_ESI() __asm PUSH ESI #define RESTORE_ESI() __asm POP ESI #else #define SAVE_ESI() #define RESTORE_ESI() #endif /*////////////////////////////////////////////////////////////////////// // The common prolog code for all DD_* functions //////////////////////////////////////////////////////////////////////*/ #define HOOKFN_PROLOG() \ /* All hook functions automatically get the same three local */\ /* variables. */\ DDEVENTINFO stEvtInfo ; /* The event information for the function*/\ DWORD dwLastError ; /* The last error value */\ REGSTATE stRegState ; /* The register state to ensure that I */\ /* restore the registers correctly */\ { \ __asm PUSH EBP /* Always save EBP explicitly. */\ __asm MOV EBP , ESP /* Move the stack. */\ __asm MOV EAX , ESP /* Get the stack pointer to calculate the*/\ /* return address and the parameters. */\ SAVE_ESI ( ) /* Save ESI in debug builds. */\ __asm SUB ESP , __LOCAL_SIZE /* Save space for the local variables.*/\ __asm ADD EAX , 04h + 04h /* Account for PUSH EBP and the */\ /* return address. */\ /* Save start of parameters on the stack.*/\ __asm MOV [stEvtInfo.dwParams] , EAX \ __asm SUB EAX , 04h /* Get back to the return address. */\ __asm MOV EAX , [EAX] /* EAX now holds the return address. */\ /* Save the return address. */\ __asm MOV [stEvtInfo.dwAddr] , EAX \ __asm MOV dwLastError , 0 /* Initialize dwLastError. */\ /* Initialize the event information. */\ __asm MOV [stEvtInfo.eFunc] , eUNINITIALIZEDFE \ __asm MOV [stRegState.dwEDI] , EDI /* Save the two registers that */\ __asm MOV [stRegState.dwESI] , ESI /* need to be saved across */\ /* function calls. */\ } /*////////////////////////////////////////////////////////////////////// // The common epilog code for all DD_* functions. iNumParams is the // number of parameters to the function that is used to restore the // stack to the proper place after the hook call. //////////////////////////////////////////////////////////////////////*/ #define HOOKFN_EPILOG(iNumParams) \ { \ SetLastError ( dwLastError ) ; /* Set the real function's last */\ /* error value. */\ __asm ADD ESP , __LOCAL_SIZE /* Add back the local variables. */\ __asm MOV EBX , [stRegState.dwEBX]/* Restore all the registers so */\ __asm MOV ECX , [stRegState.dwECX]/* that this call looks */\ __asm MOV EDX , [stRegState.dwEDX]/* identical to the intercepted */\ __asm MOV EDI , [stRegState.dwEDI]/* function. */\ __asm MOV ESI , [stRegState.dwESI] \ __asm MOV EAX , [stRegState.dwEFL] \ __asm SAHF \ __asm MOV EAX , [stRegState.dwEAX] \ RESTORE_ESI ( ) /* Restore ESI in debug builds. */\ __asm MOV ESP , EBP /* Put back ESP. */\ __asm POP EBP /* Restore the saved EBP. */\ __asm RET iNumParams * 4 /* stdcall restore of the stack */\ } /*////////////////////////////////////////////////////////////////////// // The REAL_FUNC_PRE_CALL macro needs to be placed IMMEDIATELY *BEFORE* // ANY call to the real function the hook function is handling. The // macro ensures that EDI and ESI are returned in the same condition // they were in when passed to the hook function. //////////////////////////////////////////////////////////////////////*/ #define REAL_FUNC_PRE_CALL() \ { \ __asm MOV EDI , [stRegState.dwEDI] /* Restore the real EDI. */\ __asm MOV ESI , [stRegState.dwESI] /* Restore the real ESI. */\ } /*////////////////////////////////////////////////////////////////////// // The REAL_FUNC_POST_CALL macro needs to be placed IMMEDIATELY *AFTER* // ANY call to the real function the hook function is handling. All the // register values after the real call are saved so that the hook // function epilog can return the same register values as the real // function call. //////////////////////////////////////////////////////////////////////*/ #define REAL_FUNC_POST_CALL() \ { \ __asm MOV [stRegState.dwEAX] , EAX /* Save the EAX value. */\ __asm MOV [stRegState.dwEBX] , EBX /* Save the EBX value. */\ __asm MOV [stRegState.dwECX] , ECX /* Save the ECX value. */\ __asm MOV [stRegState.dwEDX] , EDX /* Save the EDX value. */\ __asm MOV [stRegState.dwEDI] , EDI /* Save the EDI value. */\ __asm MOV [stRegState.dwESI] , ESI /* Save the ESI value. */\ __asm XOR EAX , EAX /* Zero out EAX. */\ __asm LAHF /* Load the flag values into AH*/\ __asm MOV [stRegState.dwEFL] , EAX /* Save the flag values. */\ } \ dwLastError = GetLastError ( ) ; /* Save the last error value. */\ { \ __asm MOV EAX , [stRegState.dwEAX] /* Restore EAX to its original */\ /* value. */\ /* Set the return value for */\ /* the information. */\ __asm MOV [stEvtInfo.dwRetValue] , EAX \ } /*////////////////////////////////////////////////////////////////////// // A convenient macro to fill out the event information structure //////////////////////////////////////////////////////////////////////*/ #define FILL_EVENTINFO(eFn) \ stEvtInfo.eFunc = eFn ; \ stEvtInfo.ePrePost = ePostCall ; \ stEvtInfo.dwThreadId = GetCurrentThreadId ( ) /*////////////////////////////////////////////////////////////////////// // The version 2 macros to make life MUCH, MUCH simpler to define // hook functions //////////////////////////////////////////////////////////////////////*/ // Declare this at the beginning of any hook function. // eFunc - The function enumeration value. // SynchClassType - The DDOPT_* flag value for the function class you // are processing. // bRecordPreCall - Record the entry into this function. #define HOOKFN_STARTUP(eFunc,SynchClassType,bRecordPreCall) \ HOOKFN_PROLOG ( ) ; \ if ( TRUE == DoLogging ( SynchClassType ) ) \ { \ FILL_EVENTINFO ( eFunc ) ; \ if ( TRUE == (int)bRecordPreCall ) \ { \ stEvtInfo.ePrePost = ePreCall ; \ ProcessEvent ( &stEvtInfo ) ; \ } \ } \ REAL_FUNC_PRE_CALL ( ) ; /*////////////////////////////////////////////////////////////////////// // The ending macro for a hook function. // iNuMParams - The number of parameters passed to the function. // SynchClassType - The synchronization class for the function. //////////////////////////////////////////////////////////////////////*/ #define HOOKFN_SHUTDOWN(iNumParams,SynchClass) \ REAL_FUNC_POST_CALL ( ) ; \ if ( TRUE == DoLogging ( SynchClass ) ) \ { \ stEvtInfo.ePrePost = ePostCall ; \ ProcessEvent ( &stEvtInfo ) ; \ } \ HOOKFN_EPILOG ( iNumParams ) ; /*////////////////////////////////////////////////////////////////////// // The calling convention declaration for all DD_* definitions. //////////////////////////////////////////////////////////////////////*/ #define NAKEDDEF __declspec(naked) /*////////////////////////////////////////////////////////////////////// // BIG NOTE BIG NOTE BIG NOTE BIG NOTE // The following prototypes look like cdecl functions. They are // not--they are all stdcall! The custom prolog and epilog ensure that // the correct calling convention is used! //////////////////////////////////////////////////////////////////////*/ //////////////////////////////////////////////////////////////////////// // The mandatory functions that have to be intercepted to make the // system work BOOL DD_FreeLibrary ( HMODULE hModule ) ; VOID DD_FreeLibraryAndExitThread ( HMODULE hModule , DWORD dwExitCode ) ; HMODULE DD_LoadLibraryA ( LPCSTR lpLibFileName ) ; HMODULE DD_LoadLibraryW ( LPCWSTR lpLibFileName ) ; HMODULE DD_LoadLibraryExA ( LPCSTR lpLibFileName , HANDLE hFile , DWORD dwFlags ) ; HMODULE DD_LoadLibraryExW ( LPCWSTR lpLibFileName , HANDLE hFile , DWORD dwFlags ) ; VOID DD_ExitProcess ( UINT uExitCode ) ; FARPROC DD_GetProcAddress ( HMODULE hModule , LPCSTR lpProcName ) ; //////////////////////////////////////////////////////////////////////// // The thread-specific functions HANDLE DD_CreateThread (LPSECURITY_ATTRIBUTES lpThreadAttributes , DWORD dwStackSize , LPTHREAD_START_ROUTINE lpStartAddress , LPVOID lpParameter , DWORD dwCreationFlags , LPDWORD lpThreadId ) ; VOID DD_ExitThread ( DWORD dwExitCode ) ; DWORD DD_SuspendThread ( HANDLE hThread ) ; DWORD DD_ResumeThread ( HANDLE hThread ) ; BOOL DD_TerminateThread ( HANDLE hThread , DWORD dwExitCode ) ; // The following are the CRT thread function and are handled properly as // they are __cdecl. uintptr_t DD_beginthreadex ( void * security , unsigned stack_size , unsigned ( __stdcall *start_address )( void * ) , void * arglist , unsigned initflag , unsigned * thrdaddr ) ; uintptr_t DD_beginthread ( void( __cdecl *start_address )( void * ) , unsigned stack_size , void * arglist ) ; VOID DD_endthreadex ( unsigned retval ) ; VOID DD_endthread ( void ) ; //////////////////////////////////////////////////////////////////////// // Waiting and special functions DWORD DD_WaitForSingleObject ( HANDLE hHandle , DWORD dwMilliseconds ) ; DWORD DD_WaitForSingleObjectEx ( HANDLE hHandle , DWORD dwMilliseconds , BOOL bAlertable ) ; DWORD DD_WaitForMultipleObjects( DWORD nCount , CONST HANDLE * lpHandles , BOOL bWaitAll , DWORD dwMilliseconds ) ; DWORD DD_WaitForMultipleObjectsEx( DWORD nCount , CONST HANDLE * lpHandles , BOOL bWaitAll , DWORD dwMilliseconds , BOOL bAlertable ) ; DWORD DD_MsgWaitForMultipleObjects ( DWORD nCount , LPHANDLE pHandles , BOOL fWaitAll , DWORD dwMilliseconds, DWORD dwWakeMask ) ; DWORD DD_MsgWaitForMultipleObjectsEx ( DWORD nCount , LPHANDLE pHandles , DWORD dwMilliseconds , DWORD dwWakeMask , DWORD dwFlags ) ; DWORD DD_SignalObjectAndWait ( HANDLE hObjectToSignal , HANDLE hObjectToWaitOn , DWORD dwMilliseconds , BOOL bAlertable ) ; BOOL DD_CloseHandle ( HANDLE hObject ) ; //////////////////////////////////////////////////////////////////////// // Critical-section functions VOID DD_InitializeCriticalSection(LPCRITICAL_SECTION lpCriticalSection); BOOL DD_InitializeCriticalSectionAndSpinCount ( LPCRITICAL_SECTION lpCriticalSection, DWORD dwSpinCount ); VOID DD_DeleteCriticalSection(LPCRITICAL_SECTION lpCriticalSection ) ; VOID DD_EnterCriticalSection ( LPCRITICAL_SECTION lpCriticalSection ) ; VOID DD_LeaveCriticalSection ( LPCRITICAL_SECTION lpCriticalSection ) ; DWORD DD_SetCriticalSectionSpinCount ( LPCRITICAL_SECTION lpCriticalSection, DWORD dwSpinCount ); BOOL DD_TryEnterCriticalSection ( LPCRITICAL_SECTION lpCriticalSection); //////////////////////////////////////////////////////////////////////// // Mutex functions HANDLE DD_CreateMutexA ( LPSECURITY_ATTRIBUTES lpMutexAttributes , BOOL bInitialOwner , LPCSTR lpName ) ; HANDLE DD_CreateMutexW ( LPSECURITY_ATTRIBUTES lpMutexAttributes , BOOL bInitialOwner , LPCWSTR lpName ) ; HANDLE DD_OpenMutexA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenMutexW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_ReleaseMutex ( HANDLE hMutex ) ; //////////////////////////////////////////////////////////////////////// // Semaphore functions HANDLE DD_CreateSemaphoreA ( LPSECURITY_ATTRIBUTES lpSemaphoreAttributes , LONG lInitialCount , LONG lMaximumCount , LPCSTR lpName ); HANDLE DD_CreateSemaphoreW ( LPSECURITY_ATTRIBUTES lpSemaphoreAttributes , LONG lInitialCount , LONG lMaximumCount , LPCWSTR lpName ); HANDLE DD_OpenSemaphoreA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenSemaphoreW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_ReleaseSemaphore ( HANDLE hSemaphore , LONG lReleaseCount , LPLONG lpPreviousCount ) ; //////////////////////////////////////////////////////////////////////// // Event functions HANDLE DD_CreateEventA ( LPSECURITY_ATTRIBUTES lpEventAttributes , BOOL bManualReset , BOOL bInitialState , LPCSTR lpName ) ; HANDLE DD_CreateEventW ( LPSECURITY_ATTRIBUTES lpEventAttributes , BOOL bManualReset , BOOL bInitialState , LPCWSTR lpName ) ; HANDLE DD_OpenEventA ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCSTR lpName ) ; HANDLE DD_OpenEventW ( DWORD dwDesiredAccess , BOOL bInheritHandle , LPCWSTR lpName ) ; BOOL DD_PulseEvent ( HANDLE hEvent ) ; BOOL DD_ResetEvent ( HANDLE hEvent ) ; BOOL DD_SetEvent ( HANDLE hEvent ) ; #endif // _DD_FUNCS_H |
One of the dumbest bugs I had in the first edition of DeadlockDetection was of the old "editor inheritance" variety. I created the hook functions for LoadLibraryA and LoadLibraryW and realized I also needed hook functions for LoadLibraryExA and LoadLibraryExW. Being the typical engineer, I cut and pasted the LoadLibraryA/W hook functions and edited them for the LoadLibraryExA/W functions. If you look at the HOOKFN_EPILOG macro, you'll see that it takes a parameter that is the number of function parameters. You can probably guess what happened—I forgot to change the value from 1 to 3, so calls to LoadLibraryExA/W were removing two extra items from the stack and crashing your programs!
As I looked at the code that I put in the hook functions, I realized it was basically the same thing over and over. To encapsulate the common parts, I came up with the HOOKFN_STARTUP and HOOKFN_SHUTDOWN macros. As the name implies, HOOKFN_STARTUP goes at the start of the hook function and takes care of the prolog as well any necessary pre-call logging. The parameters, in order, are the function enumeration, the DDOPT_* flag to indicate which group this function belongs to, and a Boolean flag, which if TRUE, will do pre-logging for the function. The pre-logging is for those functions such as WaitForSingleObject that would cause the potential deadlock. The HOOKFN_SHUTDOWN macro takes the number of parameters for the function and the same DDOPT_* flag passed to HOOKFN_STARTUP. Of course, to ensure I didn't make the same mistake I did with the LoadLibraryExA/W hooks, I went through and verified that the number of parameters specified in HOOKFN_SHUTDOWN was accurate.
I want to mention several other minor points about DeadlockDetection. The first is that DeadlockDetection is always active in your application, even if you suspend DeadlockDetection logging. Instead of hooking and unhooking dynamically, I leave the functions hooked and look at some internal flags to determine how the hook should behave. Keeping all functions hooked makes it easier to toggle different function logging at run time, but it adds some overhead to your application. I felt that allowing hooking and unhooking on the fly would have led to more errors in the DeadlockDetection code.
Second, DeadlockDetection hooks the functions out of a DLL when brought into your program through LoadLibrary. However, it can gain control only after that DLL's DllMain has executed, so if any synchronization objects are created or used during DllMain, DeadlockDetection can miss them.
Third, DeadlockDetection also hooks GetProcAddress and ExitProcess. The GetProcAddress hooking is there in case your program, or a third-party control you might be deadlocking against, calls GetProcAddress to find a synchronization method at run time.
I hook ExitProcess because when the application is ending, I need to unhook and shut down DeadlockDetection so that it doesn't crash or hang your program. Because there's no way to control the unload order of DLLs during program termination, you can easily get into situations in which a DLL that DeadlockDetection relies on, such as DeadDetExt, has been unloaded before DeadlockDetection itself. Fortunately, very few developers are doing major multithreaded control after the application calls ExitProcess.
For ExitProcess hooking, you'll see some special processing in DEADLOCKDETECTION.CPP. Because it's so vital to ensure that DeadlockDetection gets shut down, I forcibly hook any calls to ExitProcess, even in ignored modules. That way you don't get a surprise crash in which synchronization functions are still hooked after DeadlockDetection has already shut down.
Finally, several test programs are included with DeadlockDetection, which comes with this book's sample files. All of them are included in the main DeadlockDetectionTests solution, and all link against DEADLOCKDETECTION.DLL, so you can use them to see how DeadlockDetection operates.
|
EAN: 2147483647
Pages: 177