Windows Messages Under the Hood

In this section we're going to try to understand what is really going on when some standard Windows event is raised in your code. Consider, for example, the Paint event. You'll be familiar with the fact that you can add an event handler to this event, and with the fact that this event is normally raised when Windows detects that some portion of a form or control needs repainting. But under the hood what is happening? Windows itself has no awareness of .NET and so cannot have any awareness of the .NET events architecture. Clearly, under the hood Windows must be doing something else to make the control aware of the need to raise the Paint event. And similarly for every other event you might encounter. In fact, Windows informs the control of the need to do painting by sending it something called a Windows Message, or more commonly simply as a message. A message is a C-style struct that contains some fields. Its actual definition looks like this:

 struct MSG {    HWND     hwnd;    UINT     message;    WPARAM   wParam;    LPARAM   lParam;    DWORD    time;    POINT    pt; }; 

If you haven't programmed in C++ on Windows before, don't worry about the odd data types here. Other than POINT (the unmanaged equivalent of Point) they are all for all practical purposes just different names for integers:

• hwnd is a handle that identifies the window that the message is destined for (in this context, a window means either a form or a control).

• message is a number that identifies what the message actually is. Whereas the nature of an event is identified by which event it is, the nature of a message is determined by the value of this number. Generally speaking, you don't need to memorize which number indicates which message as symbolic constants are defined for them all (in the WinUser.h header file). For example, the constant for a paint message is always written as WM_PAINT. That for a mouse move message is WM_MOUSEMOVE, and the message that says the user has clicked on a menu is WM_COMMAND.

• wParam and lParam are simply two integers that contain more information about the message. Their meaning depends on what the message is. For example, for WM_COMMAND they will contain information about which menu item has been clicked.

• time and pt should have fairly obvious meanings - they are the time and the position of the mouse hotspot when the message was raised.

Processing Messages

So that's what a message looks like. How does Windows get the message to the application? The best way to answer that is probably with a diagram:

The thick lines with arrows indicate the flow of execution of the main user interface thread in the process. The thin dashed arrow indicates that messages are pulled off the message queue and read in the message loop.

The message queue is a data area in Windows that is used to store any messages that need to be sent to a window. The windows procedure and the message loop are both functions that must be implemented by the application that wants to display windows.

The windows procedure has the task of processing messages sent to a window. It is the function that comes closest in concept to a .NET event handler. However, where a .NET event handler is specifically geared to one event and is normally only called when that event is raised, the windows procedure is much more general purpose in nature. It is there to handle all messages - paint messages, resize messages, click messages, and so on. This means in practice that windows procedures normally end up being implemented by big switch statements, something like this:

 // This is C++ code switch (message)       // message is a UINT (unsigned int32 which // Indicates which 'event' has occurred { case WM_COMMAND:       // User has clicked on a menu item    ExecuteMenuHandler();    break; case WM_PAINT:         // Equivalent to Paint event    ExecutePaintMethod();    break; case WM_MOUSEMOVE:     // Mouse has moved    ExecuteMouseMoveHandler();    break; // etc. 

.NET is not the first API to hide the Windows message loop underneath layers of API code supposedly designed to make the developer's life easier, not only has VB always done this, but MFC does too.

The message loop is the thing that links the message queue to the Windows procedures. In the diagram we've shown the message loop as being the function that the program enters when it starts running - and with good reason. Running the message loop is what the main thread of an application normally spends most of its time doing. The message loop normally does the following:

1. Checks to see if there is a message on the queue waiting to be processed. If there isn't, then it goes to sleep and wakes up when a message appears.

2. When a message is on the queue, it reads it (popping it off the queue in the process), then calls the appropriate windows procedure, passing it the details of that message. This step is known as dispatching the message.

3. Goes back to step 1.

Note that a given thread will by default have just one message loop (although there are situations in which other loops can be added), but there is often a different windows procedure for each type of window (form or control) in the application. For example, there will be a windows procedure for all text boxes, one for all buttons, and so on. The windows procedures for standard controls are implemented by Microsoft, while those for your own controls and forms are implemented by you (or, if you are using an API such as .NET or MFC, under the hood by the API). When a message is dispatched, Windows is automatically able to make sure the correct procedure is invoked. This all happens on the same thread (the thread that executes the message loop also executes all the windows procedures). Other threads are only involved if some code inside a windows procedure explicitly calls up another thread to do some task. As I remarked in Chapter 9, a thread that is executing a message loop is normally termed a user interface thread.

The whole process normally ends when a WM_QUIT message has been received, which indicates that the application needs to terminate.

In programmatic terms, typical code for a message loop normally looks like this.

 // This is C++ code MSG msg; while (GetMessage(&msg, NULL, 0, 0)) {    TranslateMessage(&msg);    DispatchMessage(&msg); } 

If you want to see a real example of this code, just start up VS.NET, and ask it to generate a new unmanaged C++ Windows application, taking the default settings. This produces a complete working C++ application that displays a form, complete with message loop and windows procedure, and all done using raw Windows API functions. There is no class library of any kind wrapping the API calls, so you see directly what is happening. I will warn you, though, that you'll find the code looks rather more complicated than what I've presented here.

The GetMessage() API call is implemented as part of the Windows OS, and it does exactly what its name suggests - it retrieves the next message from the message queue. If there is no message, then GetMessage() has its thread put to sleep until a message appears (taking virtually no processor time while it sleeps). The details of the message retrieved are returned via the first argument to GetMessage(), which is a MSG struct of the type we saw earlier. The return value from GetMessage() is normally used as a quick way of determining if the message loop should carry on. It is zero if the message retrieved was the WM_QUIT message (which indicates the application should close down), and one otherwise. Hence the above code will always exit the while statement when a WM_QUIT message is received.

While we are inside the loop, two operations happen to the message - TranslateMessage() is an API call which performs certain changes on messages to simplify processing of text. It is TranslateMessage() that is responsible, for example, for figuring out that a WM_KEYDOWN followed by a WM_KEYUP is equivalent to a WM_CHAR (a character has been received from the keyboard). Then DispatchMessage() is where the real action happens. This API call examines the message it has been passed and then invokes the appropriate windows procedure for that message.

That's the basic principle. I should stress that what I've presented here captures the essence of what's going on but does ignore quite a few complications. For example, in order for this to all work, the application will have to inform Windows about which types of form or control it implements and where the message handlers for those controls are. (This process is called registering a window class.) I've also been loosely talking about the "next message" on the message queue without questioning which message will be retrieved first if there is more than one message on the queue. In general, GetMessage() will retrieve messages in the same order that they were posted, but there are a couple of exceptions. Most notably, if there is a WM_QUIT message on the queue, that will always be given priority over all other messages (because if you want to quit the application, there's probably not much point in the application doing anything else).

If you're wondering how messages get put on the message queue, there are two main ways. Firstly, Windows itself puts messages there when it detects things an application needs to respond to, such as mouse clicks and mouse movements. Secondly, an application can post a message to the queue by calling the PostMessage() API function.

Windows Forms and the Message Queue

Now we've seen how the message queue works for unmanaged applications, we can bring in Windows Forms and the CLR. What exactly do the System.Windows.Forms classes do that turns what is really happening (the message loop) into what your application sees (a class which is derived from Control and which raises events whenever something happens)? The answer is relatively simple in principle. Somewhere buried deep within the Sysem.Windows.Forms.Control class there will be some code that implements a windows procedure. That windows procedure will be implemented to invoke the various methods in Control and its derived classes which perform default processing. For example, in response to the WM_PAINT event, the windows procedure implemented by the NativeWindow class will call the virtual method Control.Paint(PaintEventArgs e). In addition, it will also check whether your code has added any event handlers to the Paint event of the managed Control-derived object, and if so, invoke them. Notice how the managed events arise from this process entirely as a consequence of managed code - there is no direct connection between the events you see in the Control class and the underlying Windows messages, beyond the fact that many of them happen to have been defined with similar meanings. The sequence looks a bit like the following diagram. To make the diagram more concrete I've drawn the case for a WM_PAINT message, and assumed the receiving control is a System.Windows.Forms.TextBox.

Incidentally, this sequence is the reason why some developers regard it as better practice to override Control.OnPaint() and similar methods rather than add an event handler to the Paint event. Overriding OnPaint() will give marginally better performance because it saves raising an event. However, the difference is really too small to be significant in most applications.

That accounts for the windows procedures. How about the message loop? That comes from the Application object. If you code in C++ or C#, you'll be aware that the Main() method for a Windows Forms application looks like this:

 // C# code static void Main() {    Application.Run(new Form1()); } 

The static Application.Run() method takes the form that it has been supplied, locates its windows procedure, and registers it with Windows. Then it enters a message loop (whether the code for the message loop is implemented by the Control class or by the Application class, or by some other class contained as a member field in one of these is an internal implementation detail which is not documented).

If you normally code in VB, you might not have seen the Application.Run() method. That's because the VB compiler hides even more from you than the C# compiler. If it is compiling a Windows executable program, it can automatically add this code behind the scenes. VS.NET takes advantage of this, which means you don't see the Main() method or the call to Application.Run() in your source code. But if you use ildasm.exe to examine the emitted IL, you'll see that it's there.

Leveraging the Message Loop

Now we have the basic principles under our belt let's see what implications this knowledge has for our Windows applications.

Execution Order

If you've done a lot of Windows Forms programming then the chances are that at one time or another, you've called some method which should display something on the screen, only to find that the data has only been displayed later on, not at the time that you called the method. If you don't understand how the message loop works, then this type of bug can be quite puzzling, but with our knowledge of the message loop we can now see the explanation.

In general, most painting is done because the system has posted a WM_PAINT message and allowing the windows procedure to deal with the message. Doing it this way has the advantage that if there are several paint jobs to be done, Windows can do clever things such as combining the messages on the queue, which might save some duplicated painting - something that is important since painting is one of the most processor-intensive and time-consuming tasks a Windows Forms application commonly does. Now suppose you have a list box called listBox1, and you execute this code:

 void MyMethod() {    // Some processing    listBox1.Items.Add("My new item");    // Some more intensive processing } 

What happens is that the new text gets added to some internal data structure inside the Items object (an instance of the nested class, ListBox.ObjectCollection). The list box then requires some repainting to show the new item. However, the Add() method does not do this painting. Instead, it posts a WM_PAINT message to the queue, with accompanying data that indicates exactly which bit of the list box needs repainting. Then the Add() method returns control back to the MyMethod() method which carries on performing its intensive processing. If this intensive processing lasts for several seconds, you won't see the new item appear in the list box for several seconds, because the required drawing to the screen won't take place until the WM_PAINT message is processed. In a Windows Forms application, if MyMethod() is being executed on the main thread, it must have been invoked by some event handler - that's the only way that the main thread in a Windows Forms application ever normally executes any methods: everything happens in response to an event. The main thread won't get to look at the message queue until the current event handler has completely finished. Once that happens, execution flow will transfer back into the message loop and, assuming there aren't any other higher priority messages, the WM_PAINT message will be processed and the list box will be updated on the screen.

There are two consequences of this architecture. We've just seen the first consequence, but the second consequence is more subtle - the message loop is being executed by the main UI thread, which means that it is always the main UI thread which gets to handle the messages. So even if, in the above example, MyMethod() was actually being executed on some other worker thread, the actual painting will end up taking place on the main application thread, in the Paint event handler(s). Clearly, if you are writing multithreaded applications, that information is rather important for writing correct thread synchronization code! Of course, if MyMethod() is executing on a different thread, then the actual painting might take place before MyMethod() exits, if the main thread becomes available to process messages at an earlier time. Incidentally, that the painting takes place on the main thread is normally regarded as a good thing since it's good practice to only allow one thread to deal with the user interface. The Windows Forms classes have been designed with this principle in mind, which means most of their methods are not thread safe.

The above reasoning applies to any method on the Windows Forms classes that works internally by posting a message. In general, all drawing operations work this way. So if you do anything to a control which would required it to be redrawn, or for that matter, if you call Control.Invalidate() to explicitly request repainting, then the painting won't happen until the WM_PAINT message is processed. If for some reason you do want the display to be updated immediately, then you can call the Control.Refresh() method. This method is similar to Invalidate(), but it doesn't go through the message loop. Instead it directly calls the windows procedure, passing it a WM_PAINT message. This by passes the message loop so that the painting code can execute immediately, but still means we can take advantage of things like merging invalidated regions. In that case, the above code would look like this:

 void MyMethod() {    // Some processing    listBox1.Items.Add("My new item");    listBox1.Refresh();    // Some more intensive processing. } 

You can also use the Control.Update() method, which is similar to Refresh(), but Update() just gets the control redrawn without calling Invalidate() first.

The above discussion applies to methods such as Invalidate() which internally post a message to the queue. Don't get confused between this and events. If you define your own event in a control or form, then raise the event, the event handler will be executed inline, just as for events and delegates in any other application. The message loop is not involved in this process. Remember that Windows Forms events don't have anything directly to do with the message queue.

Multithreading

One of the neatest things about the message loop is the way that is allows threads to communicate with each other in a multithreaded application. When the main UI thread picks up a message from the queue, it doesn't know or care which thread originally left the message there. All it cares about is that there's a message with some data that needs to be processed. This means that if you have worker threads in an application, then these threads can leave data or instructions for the main thread simply by posting a message to the queue - which is what we've just seen happens under the hood when a control needs repainting. This provides a convenient alternative way of communicating between threads that does not involve the thread synchronization primitives we discussed in Chapter 9. This facility is restricted: it only provides one-way communication, from a worker thread to the UI thread that is running the message loop, and it has the disadvantage that posting and processing a message involves more overhead than, for example, using a Monitor class. On the other hand, posting messages cannot block a thread, and it means you can avoid some of the subtle thread synchronization bugs that occur with traditional means of passing data between threads.

If you want to use this technique to pass data to the main thread (or if you want to get some method called on the main thread), then you will need to use the Control.BeginInvoke()/EndInvoke() methods, which we'll examine in the next section.

Another intriguing aspect of the message loop is the way that the message loop gives you many of the benefits of multithreading, but without some of the disadvantages - even just for the single UI thread. Recall that the real gain in a multithreaded program is the way that the program can smoothly context switch, as the CPU swaps between different threads, making it appear that several tasks are being executed at the same time. With a message loop, this switching between different tasks happens too. The program responds to one event, then responds to another event. Each time it has finished executing one event handler, the thread is free to perform whatever task is waiting next. This actually means that you can have the UI thread of a message loop-based application perform background tasks in a similar manner to having a background thread. Despite this, however, it's normal to use additional threads to perform background tasks in order to avoid blocking the user interface for any time. An important principle of the operation of the message loop is that processing each message should take as little time as possible, so that the computer quickly becomes free to respond to the next message, hence slow non-UI-related tasks will normally be performed by worker threads away from the message loop.

BeginInvoke()

BeginInvoke() is an incredibly useful method implemented by the Control class to provide a way of getting the main thread to do something. In terms of the way you call this method, it looks on the outside very similar to the BeginInvoke() methods that are implemented by many delegates. Like the delegate version, Control.BeginInvoke() comes as part of a method pair - there's a similar, though less useful, EndInvoke() method, which waits for the method executed by BeginInvoke() to return. However, the internal implementation could hardly be more different. BeginInvoke() called against a delegate will cause the specified method to be executed on a randomly chosen thread pool thread. Control.BeginInvoke() by contrast will execute the method on the application's main UI thread. Control.BeginInvoke() appears to work internally by posting a custom message to the message queue, this message being recognized by the Control class as a request to execute the specified method.

In terms of semantics, BeginInvoke() looks like this:

 // Definition only. IAsyncResult BeginInvoke(Delegate method, object[] args) 

The first parameter is a delegate (of any type), which indicates the method to be called. The second parameter contains an array of objects that should be passed to this method. This array will be unpacked and each element passed in - for example, if the method is expecting two parameters, you should supply an array containing two elements. A second override of BeginInvoke() does not have this argument and can be used to invoke methods that take no arguments. The IAsyncResult interface that is returned can be used in the same way as that returned from the delegate implementation of BeginInvoke(): you can monitor the status of the method, and supply it to a call to EndInvoke(). Note that, although BeginInvoke() invokes a method on the main thread, you can still think of it as an asynchronous method call. If you call BeginInvoke() from a worker thread, then the effect is very similar - the call is sent off to a different thread. If you call BeginInvoke() from the main thread, then the call still returns immediately, and the method call request is left waiting to execute as soon as the event handler that is currently executing has completed. In this chapter we'll see a couple of examples of how you can use BeginInvoke(), in both a single-threaded and a multithreaded environment.

Bear in mind that, although Control.BeginInvoke() is available, you can still call the delegate-implemented version of BeginInvoke(), if running on a thread pool thread is what you need.

It's also worth noting another method, Control.Invoke(). This has a similar operation to BeginInvoke(), but where BeginInvoke() returns immediately, Invoked() will actually block the calling thread until a the main thread has returned a result from the operation, and return the result. Invoke() can be simpler to use but is more likely to hurt performance through thread blocking, and whereas BeginInvoke() can be called from any thread, Invoke() is intended for worker threads.

Handles

Although handles are not strictly speaking anything directly to do with the message loop, now is a convenient place for a quick word about them. In Windows Forms, controls, forms, windows, and so on are represented by classes, and drawing objects work the same way. There's a Pen class, a Brush class, and so on. However, the Windows API was written many years ago, when C was the common language of choice, and object-oriented programming, classes, and C++ represented new ideas that were not yet in common use. Hence the Windows API does not use classes. Instead, objects such as windows and graphics objects are represented by handles. A handle is simply an integer that can be used to identify a resource. If you think of them as indexes into a table of data structures that the OS knows about, the table being located deep in the bowels of Windows, then you won't be far wrong. Although handles don't really come up in elementary Windows Forms programming, they are always there under the hood, and when you start to do more advanced stuff you do sometimes encounter them - and they will crop up in some of the example code later in this chapter. We also indicated earlier that the first field in the MSG structure that represents a Windows message is a handle, which identifies the window the message is intended for.

In managed code, handles are usually stored in an IntPtr, the type of choice for storing unmanaged pointers and any native integers whose size is determined by the hardware.

Accessing the Message Loop

Although I've said a lot about how the Windows Forms classes normally hide the underlying message loop from you, in fact the Control class does define a couple of methods that allow you to tap in to the message loop at a relatively low level - to the point at which you can actually alter the processing the windows procedure does in response to certain messages. One reason why you would need to do this is that, although the list of events defined in Control and derived classes is quite comprehensive, it is confined to those events relevant to common UI scenarios. There are a number of more rarely used Windows messages for which the Control class does not define any corresponding events. This list includes, for example, the various non-client messages that are related to painting and UI events outside the client area of the form. If you do, for some reason, need to handle a message for which there is no corresponding event, you will need to work at a lower level than the usual Windows Forms events. We will present an example soon which demonstrates how you can override Control.WndProc() to provide custom processing for certain Windows messages.

Idle Processing

Idle processing means that you arrange to have the main UI thread call a method known as an idle event handler when there are no messages waiting on the queue. The idle event handler does some processing, then returns control back to the message loop. Doing things this way is less flexible than real multithreading - and carries a real risk of destroying application responsiveness if you're not extremely careful, but it does save you from worrying about having multiple threads and all the associated thread synchronization issues. Doing idle processing used to be a common technique for giving some appearance of multitasking in Windows applications back in the days when writing genuinely multithreaded applications was quite difficult. Since .NET provides such extensive support for multithreaded techniques, there really is very little excuse for using idle processing techniques in a Windows Forms application. However, if you do for any reason wish to do so, then all you need to do is supply a handler to the Application.Idle event in your code.