DLL Injection: An Example

[Previous] [Next]

Let's say that you want to subclass an instance of a window created by another process. You might recall that subclassing allows you to alter the behavior of a window. To do this, you simply call SetWindowLongPtr to change the window procedure address in the window's memory block to point to a new (your own) WndProc. The Platform SDK documentation states that an application cannot subclass a window created by another process. This is not exactly true. The problem with subclassing another process's window really has to do with process address space boundaries.

When you call SetWindowLongPtr to subclass a window, as shown below, you tell the system that all messages sent or posted to the window specified by hwnd should be directed to MySubclassProc instead of the window's normal window procedure.

 SetWindowLongPtr(hwnd, GWLP_WNDPROC, MySubclassProc); 

In other words, when the system needs to dispatch a message to the specified window's WndProc, it looks up the address and then makes a direct call to WndProc. In this example, the system sees that the address of the MySubclassProc function is associated with the window and makes a direct call to MySubclassProc instead.

The problem with subclassing a window created by another process is that the subclass procedure is in another address space. Figure 22-1 shows a simplified view of how a window procedure receives messages. Process A is running and has created a window. The User32.dll file is mapped into the address space of Process A. This mapping of User32.dll is responsible for receiving and dispatching all sent and posted messages destined for any window created by any thread running in Process A. When this mapping of User32.dll detects a message, it first determines the address of the window's WndProc and then calls it, passing the window handle, the message, and the wParam and lParam values. After WndProc processes the message, User32.dll loops back around and waits for another window message to be processed.

Figure 22-1. A thread in Process B attempting to subclass a window created by a thread in Process A

Now suppose that your process is Process B and you want to subclass a window created by a thread in Process A. Your code in Process B must first determine the handle to the window you want to subclass. This can happen in a variety of ways. The example shown in Figure 22-1 simply calls FindWindow to obtain the desired window. Next, the thread in Process B calls SetWindowLongPtr in an attempt to change the address of the window's WndProc. Notice that I said "attempt." This call does nothing and simply returns NULL. The code in SetWindowLongPtr checks to see whether one process is attempting to change the WndProc address for a window created by another process and simply ignores the call.

What if the SetWindowLongPtr function could change the window's WndProc? The system would associate the address of MySubclassProc with the specified window. Then, when this window was sent a message, the User32 code in Process A would retrieve the message, get the address of MySubclassProc, and attempt to call this address. But then you'd have a big problem. MySubclassProc would be in Process B's address space, but Process A would be the active process. Obviously, if User32 were to call this address, it would be calling an address in Process A's address space, and this would probably result in a memory access violation.

To avoid this problem, you want the system to know that MySubclassProc is in Process B's address space and then have the system perform a context switch before calling the subclass procedure. Microsoft did not implement this additional functionality for several reasons:

• Applications rarely need to subclass windows created by threads in other processes. Most applications subclass windows that they create, and the memory architecture of Windows does not hinder this.
• Switching active processes is very expensive in terms of CPU time.
• A thread in Process B would have to execute the code in MySubclassProc. Which thread should the system try to use? An existing thread or a new thread?
• How would User32.dll be able to tell whether the address associated with the window was for a procedure in another process or in the same process?

Because there are no great solutions to these problems, Microsoft decided not to allow SetWindowLongPtr to change the window procedure of a window created by another process.

However, you can subclass a window created by another process—you simply go about it in a different way. The question isn't really about subclassing—it's about process address space boundaries. If you could somehow get the code for your subclass procedure into Process A's address space, you could easily call SetWindowLongPtr and pass Process A's address to MySubclassProc. I call this technique "injecting" a DLL into a process's address space. I know several ways to do this. We'll discuss each of these in turn.