Anatomy of the Tablet PC s Tablet Input Subsystem

Anatomy of the Tablet PC s Tablet Input Subsystem

An architectural view of the TIS is outlined in Figures 4-1 and 4-2. There are two main configurations under which the subsystem runs: the first is with Windows XP Tablet PC Edition, and the second is with a Windows 2000 based or Windows XP based operating system with the Tablet PC runtime libraries installed.

We ll be focusing primarily on the subsystem as it runs on Windows XP Tablet PC Edition because it s a superset of the Tablet PC runtime. But don t worry; any differences between the two configurations will be pointed out as they arise.

Figure 4-1. The Tablet PC tablet input subsystem architecture, shown running on Windows XP Tablet PC Edition

Figure 4-2. The Tablet PC tablet input subsystem architecture, shown running as the Tablet PC runtime libraries on a Windows-based operating system

Tablet Hardware

In Chapter 1, the essence of a Tablet PC was identified as a portable PC combined with a digitizer integrated with the screen, driven by Windows XP Tablet PC Edition. A digitizer (sometimes ambiguously referred to as just a tablet) in the Tablet PC Platform s view is a device that provides user input to a computer via a pointer on a flat rectangular surface. The device is able to sample the X and Y position of the pointer at regular time intervals and determine whether the pointer is active typically, this means whether the pointer is touching the tablet s surface. In most cases, the pointer is a pen stylus, although the Tablet PC Platform also recognizes a mouse as a tablet device. Indeed, a mouse is an example of a tablet device it reports X and Y position as well as whether the pointer is active.

NOTE
Placing the pointer tip on the digitizer surface is called making the pointer active. A synonym for active is down.

Chock-full of HID-y Goodness

Before the days of Tablet PC, a low-level standard already existed that directly facilitated getting input from a tablet device. Known as HID (short for Human Interface Device), the specification was initially developed to standardize communication to USB hardware such as keyboards, mice, joysticks, tablets, and just about any other device we can use to generate input for using a computer. The tablet input subsystem exclusively leverages HID devices for tablet input, and as such, the digitizers on Tablet PCs (even ones integrated with the video display) are either USB-based and HID-compliant via firmware or come supplied with a miniport driver emulating a HID interface.

NOTE
At the time of this writing, the full HID specification can be found at the USB Implementers Forum Web site, at http://www.usb.org/developers/hidpage.html, under the heading Device Class Definition for Human Interface Devices (HID).

WinTab

Some alert readers may be aware of another standard tablet input API called WinTab (http://www.pointing.com/WINTAB.HTM). It was discovered during the development of the TIS that using WinTab would prove problematic WinTab s API design puts all responsibility for functional conformance on the shoulders of the driver s implementers (third parties). Some already released WinTab drivers behavior deviated slightly from the spec, so it became difficult or impossible for the TIS to support WinTab in a generic way. Contrast this with the HID model in which the device or device driver has to specify only supported functionality (named usages) and Windows takes care of the API nuances. It was therefore decided that HID devices were the better way to go.

The HID driver, like most other Windows XP drivers, runs at the kernel level, constantly acquiring data from the tablet device and packaging it into HID format if it needs to. This allows the TIS to read input data from the device in a generic fashion.

The Center of the TIS Universe: Wisptis.exe

Arguably the most interesting piece in both Figures 4-1 and 4-2 is a process called Wisptis.exe because it s pretty much the heart and soul of the TIS. It acts like a hub between the HID driver of tablet hardware and applications, and it turns out to be responsible for realizing most of the requirements of the input system we defined earlier in the chapter. The Wisptis process performs the retrieval of input from the tablet devices, mouse emulation, detection of pen-based actions, and the dispatching of events to tablet-aware applications.

Why Wisptis.exe?

Wisptis.exe refers to WISP TIS. WISP (Windows Ink Services for Pen) was the former name for the Tablet PC Platform, and TIS well, hopefully at this point in the chapter you know what that means.

NOTE
Multiple tablet devices may be installed and used at the same time, a feature fully supported by the TIS. This is actually quite a common occurrence because, as we now know, the mouse is considered a tablet device, so if your Tablet PC has an integrated touchpad or is ever docked in a desktop scenario, or if you ve attached an external tablet to your desktop machine, from the TIS perspective there will be at least two tablet devices installed.

Getting Input from the Driver

Input is received from the digitizer via the HID driver and from the mouse via a low-level mouse hook. Data received from the digitizer is used for performing mouse emulation and detecting pen-based actions.

It s interesting to note that even though the mouse is viewed as a tablet device and a USB mouse uses a HID driver to communicate with Windows, the TIS does not get mouse data from the HID driver. Why? There are a couple of reasons. First, the mouse might not be USB-based (serial and PS/2 mice are still common) and a generic solution to reading mouse input is desirable; second, User32.dll opens the mouse driver exclusively, making it impossible for the TIS to get at the mouse directly. That s why the TIS gets the mouse input data from User32.dll via a low-level hook.

You might be thinking that a cleaner architectural model would be to integrate tablet support directly into User32.dll, and we d tend to agree with you. However, a design goal of the TIS (and a practical one at that) was to merely augment the existing OS as much as possible, rather than invade it with new code. Additionally, we ll soon see that User32 s existing message-based input architecture doesn t lend itself too well to tablet input.

Performing Mouse Emulation

The mouse cursor is controlled by the pen through one of two sets of mappings. The first set is used when the press-and-hold option (see Chapter 2) is disabled. Table 4-1 lists which pen actions will result in what mouse actions.

Table 4-1. A Simple Mapping of Pen Actions to Mouse Actions to Perform Mouse Emulation

User Pen Action	Resulting Mouse Action
In-air pen movement	Mouse movement with no buttons pressed, typically matching the pen tip location but using a hover filter
Pen touches digitizer with no barrel buttons pressed	Mouse left button pressed
Pen touches digitizer with barrel button pressed	Mouse right button pressed
Pen moves across digitizer s surface	Mouse movement with left or right button pressed, depends on which mouse button was determined to be pressed upon pen s contact with digitizer; movement matches pen tip location exactly
Pen lifted from digitizer	Mouse left or right button released, depends on which mouse button was determined to be pressed upon pen s contact with digitizer

This works pretty well in practice, with one small exception. Getting ToolTips to pop up can be pretty tough because the mouse cursor has to be completely still when hovering over, say, a toolbar button. Most people have a slight sway to the pen when they try to hold it still, so getting the mouse cursor to be motionless will typically require too much effort. Thus, a requirement for a hover filter is imposed, whose purpose is to ignore small changes in pen movement while the pen is in the air. Chapter 2 covers the hovering problem in more detail.

The second set of mappings is used when the press-and-hold option is enabled; observe that Table 4-2 is a little more complex than Table 4-1.

Table 4-2. The Mapping of Higher-Level Pen Actions to Mouse Actions to Perform Mouse Emulation when Press-and-Hold Is Active

Pen Action	Mouse Action
In-air pen movement	Mouse movement with no buttons pressed, typically matching the pen tip location but using a hover filter
Pen taps the digitizer	Mouse left button pressed, mouse left button released
Pen held down on the digitizer within press- and-hold time window and then pen lifted	Mouse right button pressed, mouse right button released
Pen held down on the digitizer and pen starts to move	Mouse left button pressed
Pen held down on the digitizer within press- and-hold time window and pen starts to move	Mouse right button pressed
Pen held down on the digitizer beyond press-and-hold time-out	Mouse left button pressed
Pen moves across digitizer s surface while held down	Mouse movement with left or right button pressed, depends on which mouse button was determined to be pressed upon pen s contact with digitizer; movement matches pen tip location exactly
Pen lifted from digitizer	Mouse left or right button released, depends on which mouse button was determined to be pressed upon pen s contact with digitizer

Notice how we need to trigger mouse actions on pen actions such as press-and-hold, tap, and drag. This is because when the pen initially touches the digitizer, the fact that press-and-hold is enabled means we don t yet know whether the user wants to perform a left button or a right button operation. You might think that the left mouse button can always get pressed on pen down, and when a press-and-hold occurs the left button would get released immediately, followed by a right button down, but that won t work well at all. Consider the following example: using everyone s favorite accessory, Notepad, select some text and right click it with the mouse. Notice how the selection stays intact and a context menu appears. If we use the pen touching the digitizer always causes a left button down method, whenever the user performs a press-and-hold it will cause a mouse left click followed by a right button down. Try doing those actions with the selection in Notepad. You ll see that the selection gets dismissed on the left click, and the right click displays the context menu with different items enabled definitely not desirable behavior.

Detecting press-and-hold, tap versus drag, and hover filtering brings us to the next bit of functionality Wisptis.exe provides: detection of pen-based actions.

Detecting Pen-Based Actions

One of the things that makes a Tablet PC so appealing is its emphasis on natural computing that is, taking human physiology into account for its input model. It s more natural for most people to use a pen than a mouse, as was discussed in Chapter 2.

Contrasting the pen with the mouse brings up an interesting difference: pens tend to be much noisier with their data. That s not really the correct term, as it implies low accuracy of data capture, but what s being referred to is how a pen generates much more subtle variance in movement than a mouse does because of the human factor. A mouse is normally at rest and gets interrupted from that state when we move it, whereas a pen that s held is at the mercy of our central nervous system s accuracy. To improve the Tablet PC user experience, therefore, it s a good idea for certain pen input to be filtered a little.

NOTE
Consider this author s observation: the difficulty of making ToolTips appear is exponentially proportional to the amount of caffeine ingested.

Figure 4-3 is a state diagram of how the detection of pen-based actions occurs. The boxes represent events that should be both mapped to mouse input and sent to tablet-aware applications.

Figure 4-3. A high-level state diagram illustrating how pen-based actions are detected

The precise algorithms for determining pen hovering and tap versus drag won t be covered here because they re a little out of this book s scope.

Dispatching Events

It s not helpful if Wisptis.exe can determine all sorts of useful pen and mouse input but can t tell anybody about it. Luckily for us, the last key function Wisptis.exe performs is client notification of input events for both mouse and tablet input.

Mouse input is easy enough to send to applications because User32.dll provides a convenient function named SendInput to automate mouse action. Tablet input, on the other hand, needs a more efficient mechanism because there s typically so much data to send. Remember that in order to have great ink, the frequency of data sampling must be high and pen handling such as pressure and tilt should be captured if possible. Both of these variables raise the required data throughput of tablet input data across processes substantially. The Windows mouse message architecture would not be able to efficiently handle the data throughput requirements of tablet input because of both the high amount of data per message and its sampling frequency. Realistically behaving ink needs to be responsive and accurate; therefore, another mechanism must be used.

Wisptis.exe communicates tablet input events to a tablet-aware application using RPC (remote procedure) calls and a shared-memory queue. A DLL running in the app s process space, InkObj.dll, receives notifications from Wisptis.exe that events are occurring, reads them from the shared-memory queue, and dispatches them to the appropriate handler in the tablet-aware app. Figure 4-4 illustrates the communication of data between Wisptis.exe and a tablet-aware application.

Figure 4-4. Wisptis.exe communicates tablet input data to a tablet-aware application through a queue to avoid losing any data.

Although tablet-unaware applications receive mouse events, it s interesting to note that tablet-aware applications receive both mouse events and tablet input events. The reason for this is simple: backward compatibility. Many existing Windows technologies rely only on mouse messages OLE drag and drop, windowless controls, and even setting the mouse cursor properly all require mouse input. We ll cover some interesting side effects of receiving two sets of events later in the chapter.

Making Sense of It All

Wisptis.exe sure does a lot of stuff, doesn t it? To help illustrate what s going on, the following pseudocode highlights the main functionality of Wisptis.exe.

// WISPTIS.EXE Pseudocode while (true) { foreach (tablet in globalTabletList) { // First, get raw pen input (down, move, up) input = GetInputDataFromHID(tablet); if (no input retrieved) continue; // See if the input is for a tablet-aware app. If // it is, dispatch it to the app's instance of InkObj.dll. app = GetTargetedApplication(input); if (app is tablet-aware) DispatchInputEventToQueue(app, input); // Now see if a higher-level action can be detected penAction = GetPenBasedAction(input, pressAndHoldMode); if (no penAction detected yet) continue; // See if action is for a tablet-aware app. If // it is, dispatch it to the app's instance of InkObj.dll. if (app is tablet-aware) DispatchInputEventToQueue(app, penAction); // Map pen action to mouse action. User32.dll will do // the "hard stuff" as far as targeting windows, posting // messages to queues, etc. mouseAction = MapPenActionToMouseAction(penAction); DispatchMouseEvents(mouseAction); } }

In essence, the data that is retrieved from the HID driver is used to determine whether the targeted application is tablet aware. If the application is tablet aware, the data is dispatched to it. The data is then processed by some code to detect pen-based actions, and if an action is detected the pen action is then dispatched to the targeted tablet-aware application, if there is one. Finally, the corresponding mouse behavior is performed.

Winlogon Desktop Support

The following applies only to Windows XP Tablet PC Edition. Notice how the diagram in Figure 4-1 is divided into left and right portions this logical division represents the Winlogon secure desktop and the Application desktop. The Winlogon desktop is the one you see at the Windows XP login screen; it operates at a high security level. When the Winlogon desktop is active (for example, the user is logging in or is unlocking his or her machine), the Tablet PC Input Panel (TIP) can be used to enter credentials. Because the Wisptis.exe process executes only in a user s session, mouse emulation is then needed to be able to use the TIP. A small DLL file, named Tpgwlnot.dll, executes in the Winlogon.exe process and performs that mouse emulation, though only with basic functionality no pen-based actions are detected. Tpgwlnot.dll also launches Wisptis.exe when a user first logs on and restarts it if needed (for example, Wisptis.exe terminates because the user ends the process or an exception occurs).

What About Ink?

So far, everything we ve covered in the Tablet PC s TIS has met the requirements of the input system we defined, except for one thing: real-time ink! Some of the pieces are there such as good data capture and throughput but rendering and storing the strokes in memory aren t. The Tablet PC Platform supports this, though not as part of the TIS. Instead, that functionality lies in the domain of ink data management, mostly the subject of the next two chapters.

Now that we conceptually understand what the tablet input subsystem does, let s actually use it to do something, shall we? It s time to write some code!