Performing Simple Recognition
Performing Simple Recognition
Some application features exploiting ink recognition need only simple results: searching a document for a text string, generating a default file name, and making application gestures such as Scratch-Out, for example. The results would become overly complex if we had to consider foreground or background reco, alternative segmentation, and the like. To solve this problem, the Tablet PC Platform provides some elegant support for obtaining recognition results in its basic form.
Recognizing Text
The easiest way to perform recognition of ink into a text string is to use the Strokes class s ToString method. The stroke objects referenced in the collection are sent to the default recognizer, the recognition results are calculated, and the highest probability results obtained are then returned as a string. All this from one function cool, huh?
Sample Application BasicReco
This first sample application demonstrates just how easy it is to obtain recognition results from a collection of ink strokes. An InkControl is used to collect and edit ink, and a menu item produces a message box containing the text string computed by the default recognizer. Figure 7-1 shows what the application looks like in action.
Figure 7-1. The BasicReco sample application shows just how simple performing recognition can be.
BasicReco.cs
//////////////////////////////////////////////////////////////////// // // BasicReco.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program demonstrates the basic recognition capabilities of // the Tablet PC Platform SDK. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private InkControl inkCtl; // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed by analyzing // the number of items in the Recognizers collection Recognizers recognizers = new Recognizers(); if (recognizers.Count == 0) { MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } else { Application.Run(new frmMain()); } } // Main form setup public frmMain() { SuspendLayout(); // Create the main menu Menu = new MainMenu(); MenuItem miFile = new MenuItem("&File"); Menu.MenuItems.Add(miFile); MenuItem miExit = new MenuItem("E&xit"); miExit.Click += new EventHandler(miExit_Click); Menu.MenuItems[0].MenuItems.Add(miExit); MenuItem miReco = new MenuItem("&Recognize!"); miReco.Click += new EventHandler(miReco_Click); Menu.MenuItems.Add(miReco); // Create and place all of our controls inkCtl = new InkControl(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); // Configure the form itself ClientSize = new Size(368, 232); Controls.AddRange(new Control[] { inkCtl }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "BasicReco"; ResumeLayout(false); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Handle the "Exit" menu item being clicked private void miExit_Click(object sender, EventArgs e) { Application.Exit(); } // Handle the Recognize! menu item being clicked private void miReco_Click(object sender, EventArgs e) { // Show the results via the Strokes.ToString method MessageBox.Show(this, inkCtl.InkOverlay.Ink.Strokes.ToString(), "Results"); } } The source listing contains almost nothing new, except for one line of code found in the miReco_Click event handler:
// Show the results via the Strokes.ToString method MessageBox.Show(this, inkCtl.InkOverlay.Ink.Strokes.ToString(), "Results");
The Strokes.ToString method does a great job of providing the reco results with the highest confidence using the default recognizer. You should use Strokes.ToString with some restraint, however, because it implements recognition in an end-to-end fashion; that is to say, the method synchronously performs recognition from start to finish each time it is called. No results are cached, and no computation occurs in the background therefore, be advised that the time taken for this call to execute can be considerable with even a few sentences of ink.
Another trade-off made when using the Strokes.ToString method is that no other information about the result besides the highest probability string can be obtained; alternates and confidence information get lost in the process. But if you don t need this information maybe you re writing a forms-based application or want to include text on the clipboard or in drag-and-drop operations using Strokes.ToString can mean that adding recognition to your application takes just one line of code!
Recognizing Application Gestures
To recognize ink strokes as application gestures, we turn to the InkCollector and InkOverlay classes. Results of recognized application gestures are obtained through an event-based model rather than a synchronous query model like Strokes.ToString. The reason for this is that application gestures represent commands, which are typically received from the keyboard, mouse, or user-interface elements such as menu and toolbar button clicks, all of which take the form of events.
Specifying Application Gestures to Recognize
The InkCollector and InkOverlay classes provide application gesture recognition almost as simply as Strokes.ToString does. Recall from Chapter 4 that we can request the Gesture event to fire when an application gesture is recognized, optionally followed by a Stroke event (this depends on the value of the Cancel property of the InkCollectorGestureEventArgs object: if it is set to true, a Stroke event will fire; if it is false, the Stroke event will not fire). Setting the InkCollector or InkOverlay instance s CollectionMode property to either CollectionMode.InkAndGesture or CollectionMode.GestureOnly activates the recognition of application gestures.
The CollectionMode.GestureOnly collection mode tells the InkCollector or InkOverlay to perform gesture recognition after a timeout has occurred from the last stroke drawn, allowing for multiple strokes to form one application gesture. The Gesture event will always be fired in this collection mode, regardless of the confidence level of the results.
The CollectionMode.InkAndGesture collection mode indicates a hybrid inking mode: the recognizer attempts to interpret every ink stroke as an application gesture. If a recognition result is computed, the Gesture event is fired. If no results are computed or if the Gesture event is cancelled by setting its Ink CollectorGestureEventArgs s Cancel property to true, the Stroke event is fired. This collection mode therefore yields a best-guess approach for ink strokes. When using this collection mode, the recognizable application gestures can comprise only a single stroke.
Once the collection mode has been set on the InkCollector or InkOverlay, the class s SetGestureStatus method is typically called to activate detection for the various application gestures requested. This is done to keep the set of recognized gestures as small as possible, thereby increasing accuracy results (the smaller the set of possibilities, the more likely the results will be correct). Values in the ApplicationGesture enumeration, listed in Tables 7-1, 7-2, and 7-3, identify the application gestures to be recognized.
Table 7-1 lists the application gestures for which there are recommended actions. These are operations for which an application gesture type has a well-defined behavior and hence should be considered a UI standard. Microsoft discourages deviating from this behavior if possible.
Gesture Shape | ApplicationGesture Value | Action | Notes |
| ScratchOut | Erase content | Must contain at least three horizontal back-and-forth movements. |
| Left | Backspace | Horizontal line to the left. |
| Right | Space | Horizontal line to the right. |
| UpRightLong | Tab | Up and then right, two to four times as long as the up portion; should form a right angle. |
| DownLeftLong | Enter | Down and then left, two to four times as long as the down portion; should form a right angle. |
| DownRightLong | Space | Down and then right, two to four times as long as the down portion; should form a right angle. |
| UpDown | Undo | Vertical line, up and then down, as straight and thin as possible |
| LeftRight | Cut | Horizontal line, left and then right, as straight and thin as possible. |
| RightLeft | Copy | Horizontal line, right and then left, as straight and thin as possible. |
| Curlicue | Cut | Drawn at an angle from lower left to upper right, starting on the word to cut |
| DoubleCurlicue | Copy | Drawn at an angle from the lower left to the upper right, starting on the word to copy. |
| DoubleCircle | Paste | Two circles overlapping each other, drawn with one stroke. |
| SemiCircleLeft | Undo | Drawn from the right to the left, with the two ends of the arc lying on the same horizontal line. |
| SemiCircleRight | Redo | Drawn from the left to the right, with the two ends of the arc lying on the same horizontal line. |
| Tap | Left click | Must be short stroke with minimal area covered. |
| DoubleTap | Left double click | Two taps as close to each other as possible. This multistroke gesture is not supported by InkAndGesture collection mode. |
| RightUp | Input Method Editor (IME) convert | Right and then up with same length as the right portion; should form a right angle. |
Table 7-2 contains the application gestures for which suggested actions should be taken. These are operations for which behavior is application-specific although predictable by the context in which they re used. In other words, the outcome is descriptively similar between applications, but the exact functionality is application-specific.
Gesture shape | ApplicationGesture Value | Action | Notes |
| Triangle | Insert | Must have the top pointing up |
| Square | Action item | Must start at the upper left corner; can be drawn with one or two strokes |
| Star | Action item | Must have exactly five points |
| Check | Check a box or check off a list item | Must have the upward stroke two to four times as long as the smaller downward stroke |
| ChevronUp | Paste, insert | Must have both sides be of equal length and have a sharp-angled point |
| ChevronDown | Insert | Must have both sides be of equal length and have a sharp-angled point |
Table 7-3 lists the application gestures that are application-specific. These gestures can invoke commands that are specific to the application receiving them. There is no standard behavior associated with these gestures, and they can be used without restriction.
Gesture shape | ApplicationGesture Value | Notes |
| Circle | Must start and end at the topmost point. |
| ChevronLeft | Must have both sides be of equal length and have a sharp-angled point. |
| ChevronRight | Must have both sides be of equal length and have a sharp-angled point. |
| ArrowUp | Must be drawn using either one or two strokes. For a single stroke, a full triangle-shaped head isn t supported, and for two strokes, one stroke must be the line and the other must be the arrowhead. |
| ArrowDown | Must be drawn using either one or two strokes. For a single stroke, a full triangle-shaped head isn t supported, and for two strokes, one stroke must be the line and the other must be the arrowhead. |
| ArrowLeft | Must be drawn using either one or two strokes. For a single stroke, a full triangle-shaped head isn t supported, and for two strokes, one stroke must be the line and the other must be the arrowhead. |
| ArrowRight | Must be drawn using either one or two strokes. For a single stroke, a full triangle-shaped head isn t supported, and for two strokes, one stroke must be the line and the other must be the arrowhead. |
| Up | Vertical line upward. |
| Down | Vertical line downward. |
| UpLeft | Up and then left with same length as the up portion; should form a right angle. |
| UpRight | Up and then right with same length as the up portion; should form a right angle. |
| DownLeft | Down and then left with same length as the down portion; should form a right angle. |
| DownRight | Down and then right with same length as the down portion; should form a right angle. |
| LeftUp | Left and then up with same length as the left portion; should form a right angle. |
| LeftDown | Left and then down with same length as the left portion; should form a right angle. |
| RightDown | Right and then down with same length as the right portion; should form a right angle. |
| DownUp | Vertical line, down and then up, as straight and thin as possible. |
| Exclamation | Draw the dot soon after drawing the line and place it close to the line. This multistroke gesture is not supported by InkAndGesture collection mode. |
In addition to the entries listed in the preceding tables, two other values in the ApplicationGesture enumeration are worth mentioning here: ApplicationGesture.AllGestures and ApplicationGesture.NoGesture. ApplicationGesture.All Gestures is used with the SetGestureStatus method as a convenient shortcut to turn recognition for all application gestures on or off. ApplicationGesture.NoGesture is passed to the Gesture event when the application gesture recognizer believes there is no good result for the strokes the user entered.
Receiving Application Gesture Events
The Gesture event s InkCollectorGestureEventArgs parameter provides a list of the application gestures that were detected by the gesture recognizer, along with their confidence and hot point. The confidence amount is specified with the RecognitionConfidence enumeration, whose values are listed in Table 7-4. The hot point of the gesture is the point in ink coordinates that the user targeted while drawing the gesture. The type of application gesture defines where the specific hot point is located in relation to the gesture shape.
Value | Description |
Strong | Computed results are likely to be correct |
Intermediate | Likelihood of correct results is fair |
Poor | Computed results could likely be incorrect |
Sample Application GesturePad
This next sample application shows how gesture recognition is achieved using InkCollector s Gesture event. The sample implements a crude interface to edit text in a TextBox Cut, Copy, Paste, Delete, cursor navigation, Tab, Backspace, and Return.
GesturePad.cs
//////////////////////////////////////////////////////////////////// // // GesturePad.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program demonstrates how to handle gesture events from the // InkCollector class to implement gesture-based editing // functionality. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; public class frmMain : Form { private TextBox txtPad; private Panel pnlInput; private Button btnClose; private InkCollector inkCollector; // Entry point of the program [STAThread] static void Main() { Application.Run(new frmMain()); } // Main form setup public frmMain() { SuspendLayout(); // Create and place all of our controls txtPad = new TextBox(); txtPad.AcceptsReturn = true; txtPad.AcceptsTab = true; txtPad.Location = new System.Drawing.Point(8, 8); txtPad.Multiline = true; txtPad.ScrollBars = ScrollBars.Both; txtPad.Size = new System.Drawing.Size(280, 168); pnlInput = new Panel(); pnlInput.BackColor = Color.White; pnlInput.BorderStyle = BorderStyle.Fixed3D; pnlInput.Location = new System.Drawing.Point(96, 184); pnlInput.Size = new System.Drawing.Size(104, 72); btnClose = new Button(); btnClose.Location = new System.Drawing.Point(208, 232); btnClose.Text = "Close"; btnClose.Click += new System.EventHandler(btnClose_Click); ClientSize = new System.Drawing.Size(296, 266); Controls.AddRange(new Control[] { btnClose, pnlInput, txtPad}); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "GesturePad"; ResumeLayout(false); // Create an InkCollector object and put it in ink-and-gesture // collection mode inkCollector = new InkCollector(pnlInput.Handle); inkCollector.CollectionMode = CollectionMode.InkAndGesture; // Let the ink collector know which gestures we'd like // to have recognized inkCollector.SetGestureStatus( ApplicationGesture.Scratchout, true); inkCollector.SetGestureStatus( ApplicationGesture.Left, true); inkCollector.SetGestureStatus( ApplicationGesture.Right, true); inkCollector.SetGestureStatus( ApplicationGesture.UpRightLong, true); inkCollector.SetGestureStatus( ApplicationGesture.DownLeftLong, true); inkCollector.SetGestureStatus( ApplicationGesture.DownRightLong, true); inkCollector.SetGestureStatus( ApplicationGesture.LeftRight, true); inkCollector.SetGestureStatus( ApplicationGesture.RightLeft, true); inkCollector.SetGestureStatus( ApplicationGesture.ChevronDown, true); inkCollector.SetGestureStatus( ApplicationGesture.ArrowUp, true); inkCollector.SetGestureStatus( ApplicationGesture.ArrowDown, true); inkCollector.SetGestureStatus( ApplicationGesture.ArrowLeft, true); inkCollector.SetGestureStatus( ApplicationGesture.ArrowRight, true); // Hook up the gesture and stroke event handlers to inkCollector inkCollector.Gesture += new InkCollectorGestureEventHandler(inkCollector_Gesture); inkCollector.Stroke += new InkCollectorStrokeEventHandler(inkCollector_Stroke); // We're now set to go, so turn on gesture collection inkCollector.Enabled = true; } // InkCollector gesture event handler private void inkCollector_Gesture(object sender, InkCollectorGestureEventArgs e) { // If we're pretty sure about the gesture, perform the // operation - otherwise do nothing (to avoid doing // something wrong and frustrating the user!) if ((e.Gestures[0].Confidence == RecognitionConfidence.Strong) (e.Gestures[0].Confidence == RecognitionConfidence.Intermediate)) { switch (e.Gestures[0].Id) { case ApplicationGesture.Scratchout: // Erase txtPad.Focus(); SendKeys.Send("{DELETE}"); break; case ApplicationGesture.Left: // Backspace txtPad.Focus(); SendKeys.Send("{BACKSPACE}"); break; case ApplicationGesture.Right: case ApplicationGesture.DownRightLong: // Space txtPad.Focus(); SendKeys.Send(" "); break; case ApplicationGesture.UpRightLong: // Tab txtPad.Focus(); SendKeys.Send("{TAB}"); break; case ApplicationGesture.DownLeftLong: // Enter txtPad.Focus(); SendKeys.Send("{ENTER}"); break; case ApplicationGesture.LeftRight: // Cut txtPad.Cut(); break; case ApplicationGesture.RightLeft: // Copy txtPad.Copy(); break; case ApplicationGesture.ChevronDown: // Paste txtPad.Paste(); break; case ApplicationGesture.ArrowUp: // Cursor up txtPad.Focus(); SendKeys.Send("{UP}"); break; case ApplicationGesture.ArrowDown: // Cursor down txtPad.Focus(); SendKeys.Send("{DOWN}"); break; case ApplicationGesture.ArrowLeft: // Cursor left txtPad.Focus(); SendKeys.Send("{LEFT}"); break; case ApplicationGesture.ArrowRight: // Cursor right txtPad.Focus(); SendKeys.Send("{RIGHT}"); break; default: // We don't do anything otherwise break; } } // Make sure the stroke gets deleted e.Cancel = false; pnlInput.Invalidate(); } // InkCollector stroke event handler private void inkCollector_Stroke(object sender, InkCollectorStrokeEventArgs e) { // We never want to keep a stroke - this effectively means that // no gesture was recognized so we'll no-op. e.Cancel = true; pnlInput.Invalidate(); } // Button close clicked handler private void btnClose_Click(object sender, System.EventArgs e) { Application.Exit(); } } After creating an InkCollector instance, the application sets its collection mode to enable gesture recognition and then repeatedly calls the SetGestureStatus method to specify which gestures it s interested in. The CollectionMode.InkAndGesture mode is used because of the immediate response resulting from gesture recognition. The CollectionMode.GestureOnly mode requires a timeout to elapse once a gesture is drawn that cannot be altered, resulting in a rather painful waiting period while the user performs repeated operations such as cursor navigation. By using the ink-and-gesture mode, we get timely Gesture event firing, although we give up multistroke application gesture recognition; fortunately, however, most application gestures are formed by only a single stroke.
The Gesture event handler receives the computed results in the InkCollectorGestureEventArgs object, whose Gestures property is an array containing the computed results ordered by highest to lowest confidence. Because our application performs editing operations, only strong and intermediate confidence gestures are processed to avoid performing an unexpected action on the user s behalf:
// If we're pretty sure about the gesture, perform the // operation - otherwise do nothing (to avoid doing // something wrong and frustrating the user!) if ((e.Gestures[0].Confidence == RecognitionConfidence.Strong) (e.Gestures[0].Confidence == RecognitionConfidence.Intermediate)) { .... It is entirely possible to have the Gesture event fired with ApplicationGesture.NoGesture and a confidence level of RecognitionConfidence.Strong. This occurs when the application gesture recognizer is certain that the strokes do not look like any active application gesture, and the collection mode is CollectionMode.GestureOnly.
A Stroke event handler is provided because of the ink-and-gesture collection mode the application uses. This event gets fired when no gesture is recognized and also when the Gesture event sets the Cancel property of the InkCollectorGestureEventArgs to true. To avoid having an ink stroke added to the InkCollector s Ink object, the application s Stroke event sets the Cancel property of the InkCollectorStrokeEventArgs to true.
Erasing Ink with the Scratch-Out Gesture
The Windows Journal, Tablet PC Input Panel, and Sticky Notes applications all support erasing ink with the ApplicationGesture.Scratchout gesture, akin to scribbling out the ink as one would on physical paper. Implementing this functionality is rather straightforward, although it is worthwhile pointing out a couple of issues.The general algorithm for performing Scratch-Out erasing is as follows: for an application gesture of type ApplicationGesture.Scratchout with intermediate or greater confidence level, hit-test using the bounding box of the Scratch-Out stroke with a 60 percent tolerance. If no strokes other than the Scratch-Out are hit, keep the Scratch-Out stroke by setting the InkCollectorGestureEventArgs s Cancel property to true. If one or more strokes other than the Scratch-Out are hit, delete them including the Scratch-Out from the document. Remember to take into account the granularity of selectable objects when hit-testing and deleting if it s appropriate for your application.
If your application supports undo, a Scratch-Out operation resulting in deletion should create two undo units: the first is the deletion of the hit object or objects including the Scratch-Out stroke, and the second is the addition of the Scratch-Out stroke to the document. Adding the second undo unit reduces user frustration if a stroke results in unintended Scratch-Out erasing.
EAN: 2147483647
Pages: 73