Using the Recognition Classes
Using the Recognition Classes
While certainly convenient, the Strokes class s ToString method does have some shortcomings. Perhaps the most obvious is that the recognizer used to perform recognition cannot be specified; this means that only the default language of the system can be recognized. Another problem with the ToString recognition method is that no alternative results are provided in the event that the recognizer wasn t right the first time nor is any other information provided, such as confidence level, stroke baseline, or associated strokes, for that matter. And lastly, the entire recognition computation is performed synchronously during the ToString method call, which can take up to several seconds depending on the amount of ink being recognized.
The Tablet PC Platform provides a set of recognition classes specifically designed for these issues while maintaining a simple design indicative of the rest of the Tablet PC Platform. The usage model for performing recognition with these classes is quite simple:
Obtain a recognizer to use
Initiate a recognition session with the recognizer
Provide the strokes to recognize to the recognizer
Collect the computed results
Let s take a closer look at each of these steps.
Obtaining a Recognizer to Use
The first step in performing ink recognition is to obtain a reference to an instance of the Recognizer class. A Recognizer instance is the embodiment of a recognizer installed in the system; it therefore has properties reflecting the languages supported, its specific recognition capabilities, and name and vendor identification.
The Recognizers class encapsulates a collection of Recognizer objects, and the contents of the collection reflect the recognizers that are currently installed in the system. In this manner, the Recognizers collection is analogous to the Tablets collection we learned about in Chapter 4. An instance of the Recognizers class is obtained using the new operator and is enumerable with standard C# means; the only non-collection-related method on the class is GetDefaultRecognizer, which returns the default recognizer.
A recognizer s specific capabilities are obtained through a Recognizer instance s Capabilities property, a bitflag collection whose values are found in the RecognizerCapbilities enumeration, described in Table 7-5.
RecognizerCapabilities enum Value | Description |
ArbitraryAngle | The recognizer supports text written at any angle. |
BoxedInput | The recognizer supports boxed input, meaning characters are entered in a box. |
CharacterAutoCompletionInput | The recognizer supports character auto-complete. |
DownAndLeft | The recognizer supports text flow as in Asian languages. |
DownAndRight | The recognizer supports text flow as in the Chinese language. |
FreeInput | The recognizer supports free input, meaning no guide is needed. |
Lattice | The recognizer can return a lattice object, which is not used in the managed API. |
LeftAndDown | The recognizer supports text flow as in Hebrew and Arabic languages. |
LinedInput | The recognizer supports lined input, meaning text is written as it is on lined paper. |
Object | Objects rather than text are recognized. |
RightAndDown | The recognizer supports text flow as in Western and Asian languages. |
Initiating a Recognition Session
Once a Recognizer instance has been obtained, a recognizer context (also referred to as a reco context) must be created to perform the recognition operation. This object represents a session of recognizer functionality; associated with it are the ink strokes to recognize, parameters such as the recognition mode, preferred words, content hints, and any recognition results as they are computed. Calling a Recognizer s CreateRecognizerContext method creates a RecognizerContext object.
The Recognition Mode
The reco context s RecognitionFlags property is used to control the recognition mode. This term refers to how the recognizer will treat the ink and compute the recognition results. The RecognitionFlags property is of the RecognitionModes enumeration type, the values for which are listed in Table 7-6.
Value | Description |
Coerce | The recognizer will force the result to match the reco context s factoid. |
None | No recognition mode (default). |
TopInkBreaksOnly | Multiple segmentation results will not be returned only the segmentation with highest confidence is computed. |
WordMode | The recognizer will treat all the ink as a single word. |
The description for the RecognitionModes.Coerce flag in Table 7-6 refers to a factoid. Briefly, a factoid is a property on a reco context that describes the content of the strokes being recognized, thereby increasing accuracy. (For example, an e-mail address and a phone number are considered factoids.) If RecognitionModes.Coerce is not specified, the result returned might not match the factoid if the recognizer has a high enough confidence level for example, if the factoid is an e-mail address but the recognizer is sure the ink is a phone number, it will return a phone number. Specifying the RecognitionModes.Coerce flag ensures the results returned will match the factoid.
The RecognitionModes.TopInkBreaksOnly flag relates to the together segmentation issue covered earlier. Without this flag, the recognizer can compute different segmentations for the ink, so together, to get her, and to gather might be returned along with variations of each word, such as colatter, to got hir, and to gopher. Using RecognitionModes.TopInkBreaksOnly ensures that the recognizer will compute and return only one kind of segmentation. This helps performance and can simplify the user interface required for correcting the results.
RecognitionModes.WordMode tells the recognizer to treat all the ink it will recognize as a single word. This treatment improves accuracy when a user is required to enter only a single word of ink or when the application employs a Little Ink model. Recall the Big Ink vs. Little Ink discussion from Chapter 5, in which individual words being grouped into separate Ink objects is known as Little Ink, and all ink residing in one Ink object is known as Big Ink.
Prefix and Suffix Strings
The recognizer context properties Prefix and Suffix are used to specify the text preceding and following the content to recognize. They can be useful when building Tablet PC Input Panel style functionality because results adjacent to the word to recognize can be provided to the recognizer, and they are also useful in conjunction with the RecognitionModes.WordMode mode in providing proper context to the recognizer.
The Character Auto-Complete Mode
The character auto-complete mode (sometimes referred to as the CAC mode) is useful in conjunction with Tablet PC Input Panel style functionality for languages containing words that are made up of multiple segments, such as Chinese, Japanese, and Korean. As the user writes segments of a word, the recognizer will compute best-match whole word results rather than results for only the segments that have been entered up to that point. This allows for the provision of a user interface from which the user can choose from a list of completed possibilities, streamlining the text input process.
The RecognizerContext class s property CharacterAutoCompletion specifies the character auto-complete mode that the recognizer should use. Its type is of the RecognizerCharacterAutoCompletionMode enumeration; Table 7-7 lists the available modes.
Value | Description |
Full | Recognition occurs as if all strokes have been input (default). |
Prefix | Recognition occurs for partial word with strokes input in a known order. |
Random | Recognition occurs for partial word with strokes input in random order. |
The Full mode indicates that the recognizer won t perform character auto-completion. The Prefix mode indicates to the recognizer that auto-complete results should be computed by assuming the strokes were input in an order consistent with the correct way of writing the language. The Random value indicates that the order in which the strokes were input is random, and it happens to be the most computationally intensive. When using the character auto-complete functionality of a recognizer, it is recommended that a two-pass approach be implemented the first pass displays results to the user from the Prefix mode because it is quicker, and the second pass displays more results with the Random mode because it is more accurate.
NOTE
Using CAC functionality requires a recognizer guide be set on the reco context. We ll learn about recognizer guides later in the chapter, but for now they can be considered a hint to the recognizer about the layout of the strokes.
Supplying Strokes to the Recognizer
Now that a recognizer context object has been created and configured, it is ready to begin recognizing ink data. The reco context needs to know what ink it should perform recognition on, which you indicate by setting its Strokes property.
Earlier we talked about partial recognition the ability of the recognizer to incrementally perform recognition as ink is added to or removed from the context s Strokes collection. Partial recognition can reduce computation overhead significantly because the entire reco operation does not need to start over from the beginning when the user draws a new stroke or deletes one. If a completely new Strokes collection is assigned to the reco context s Strokes property, however, the recognition state is reset; this results in the entire recognition operation restarting.
If a recognizer does not support partial recognition, a special method must be called before trying to obtain any recognition results from the reco context. The EndInkInput method tells the recognizer that no more strokes will be added to or removed from the reco context, declaring it safe to perform the recognition operation. No result will be returned if EndInkInput is not called when using a recognizer that doesn t support partial recognition and an attempt is made to obtain recognition results.
Now let s restate the concepts of the past couple of paragraphs to make sure they re clear: if a recognizer supports partial recognition, when a reco context s Strokes collection is modified with Strokes.Add and/or Strokes.Remove, the reco results are incrementally computed. These results can be obtained at any time; there is no need to call EndInkInput. However, if a recognizer does not support partial recognition, once the reco context has been given the ink to recognize, the EndInkInput method must be called before reco results can be obtained. Subsequent modification of the Strokes collection with Strokes.Add and/or Strokes.Remove will result in a run-time error when recognition is attempted; the collection must be reassigned to an entirely new instance for new results to be obtained.
So if a recognizer supports partial recognition, is there ever a need to call EndInkInput? Yes, because querying for recognition results before calling EndInkInput does not guarantee the recognizer will provide the most accurate results or the greatest number of them. The advantage in not calling EndInkInput is that much less computation takes place over the reco context s lifetime because the computation is all incremental. Once EndInkInput is called, the reco context s Strokes collection must be reassigned to a new collection for more recognition to occur; this results in the entire reco operation starting over.
Unfortunately, there is no easy way to query a recognizer to see whether it supports partial recognition. However, we ll discuss a manner in which partial recognition can be computed later in the chapter.
Getting Results I: Easy Synchronous Recognition
Once a RecognizerContext object has been created and set up with the ink data to recognize, we request that the recognition operation be performed and then we obtain the results. We ve discussed that the results of the recognition operation can be requested either synchronously or asynchronously; let s first take a look at the synchronous case.
The Recognize method of the RecognizerContext class is used to obtain recognition results synchronously. The method returns a RecognitionResult object, which is an encapsulation of the result data and includes alternates, confidence, and stroke association. The out parameter is the RecognitionStatus value, which is effectively a success code indicating any problems that the application might have encountered during the recognition operation.
NOTE
Another term for an alternative result computation is recognition alternate, or just alternate.
Sample Application IntermediateReco
This sample application performs identically to the BasicReco application presented earlier in the chapter, but it s implemented using the recognition classes. The user enters some ink and chooses an application menu item to trigger recognition; a message box containing the recognition results is then displayed. The application uses the default recognizer for the recognition, and the recognition operation is performed all at once and in a synchronous fashion exactly like the Strokes.ToString method.
IntermediateReco.cs
//////////////////////////////////////////////////////////////////// // // IntermediateReco.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program shows how to perform recognition using the // recognizer classes in a one-shot synchronous fashion. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private InkControl2 inkCtl; private static Recognizers recognizers = new Recognizers(); // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed by analyzing // the number of items in the Recognizers collection if (frmMain.recognizers.Count > 0) { Application.Run(new frmMain()); } else { // None are, so display error message and exit MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } // Main form setup public frmMain() { SuspendLayout(); // Create the main menu Menu = new MainMenu(); MenuItem miFile = new MenuItem("&File"); Menu.MenuItems.Add(miFile); MenuItem miExit = new MenuItem("E&xit"); miExit.Click += new EventHandler(miExit_Click); Menu.MenuItems[0].MenuItems.Add(miExit); MenuItem miRecognize = new MenuItem("&Recognize!"); miRecognize.Click += new EventHandler(miRecognize_Click); Menu.MenuItems.Add(miRecognize); // Create and place all of our controls inkCtl = new InkControl2(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); // Configure the form itself ClientSize = new Size(368, 232); Controls.AddRange(new Control[] { inkCtl }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "IntermediateReco"; ResumeLayout(false); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Handle "Exit" menu item private void miExit_Click(object sender, EventArgs e) { Application.Exit(); } // Handle Recognize! menu item private void miRecognize_Click(object sender, EventArgs e) { // Obtain the default recognizer Recognizer recognizer = frmMain.recognizers.GetDefaultRecognizer(); // Create a reco context, and add ink to it RecognizerContext recoCtxt = recognizer.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; recoCtxt.EndInkInput(); // Perform the recognition on the strokes RecognitionStatus recoStatus; RecognitionResult recoResult = recoCtxt.Recognize(out recoStatus); if (recoStatus == RecognitionStatus.NoError) { // Show the results MessageBox.Show(this, recoResult.TopString, "Results"); } else { // Problem; display error message MessageBox.Show(this, recoStatus.ToString(), "Error"); } } } The application begins by checking whether any recognizers are installed in the system we do this simply by checking the number of recognizer objects found in the Recognizers collection. If there are none, which is likely the case when running the sample on non Windows XP Tablet PC Edition operating systems (discussed in Chapter 3), the application exits. If at least one recognizer is installed, the application creates a form with an InkControl along with a menu containing the Recognize! item, just as in the BasicReco sample.
What s different between this sample and the BasicReco application is the handler for the Recognize! menu item:
// Obtain the default recognizer Recognizer recognizer = frmMain.recognizers.GetDefaultRecognizer(); // Create a reco context, and add ink to it RecognizerContext recoCtxt = recognizer.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; recoCtxt.EndInkInput(); // Perform the recognition on the strokes RecognitionStatus recoStatus; RecognitionResult recoResult = recoCtxt.Recognize(out recoStatus); if (recoStatus == RecognitionStatus.NoError) { ... Here, the default recognizer is obtained and a recognizer context is created. The reco context is told what ink it should perform recognition on by setting its Strokes property to reference all strokes in the InkControl s Ink object. Next the EndInkInput method is called on the reco context to signify that no more ink will be added to the Strokes collection. This makes it safe for the next step, calling the Recognize method on the reco context. If the recognition status indicates the method completed successfully, the returned RecognitionResult object s highest confidence result is displayed from the TopString property.
Getting Results II: Electric Boogaloo (a.k.a. Harder Synchronous Recognition)
Performing recognition all at once is inefficient for most applications. Users will be frustrated if they invoke a command that triggers recognition and they re forced to wait for multiple seconds while the results are obtained synchronously. This problem can be overcome if the recognizer supports partial recognition and we keep the reco context in the loop regarding ink getting added to or removed from the document.
By keeping the reco context up-to-date on the strokes it will recognize, we gain the advantage of the recognizer incrementally performing the recognition operation. Thus, once results are finally requested, the computation operation is that much nearer to completion than it would be when starting from scratch. To let the reco context know whenever ink is added or removed, its Strokes collection should be modified accordingly. The Ink class s InkAdded and InkDeleted events are great for this purpose.
Sample Application AdvancedReco
This sample application shows how a recognizer context can be retained throughout the life of an ink document, keeping it up-to-date with the ink strokes in the document. In addition, instead of using only the default recognizer for recognition, the application fills the main menu with all installed recognizers. Users can then choose what recognizer they want to use when a recognition operation is performed.
The application creates the recognition context when a recognizer is chosen, and ink is added to or removed from the reco context s Strokes collection as it is added to and removed from the InkControl s Ink object. As you ll see, this greatly helps performance: by keeping the reco context up-to-date with the state of the ink, recognition occurs in the background while the user adds or deletes strokes, or pauses for some reason.
AdvancedReco.cs
//////////////////////////////////////////////////////////////////// // // AdvancedReco.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program shows how to enumerate and use the installed // recognizers in a synchronous manner. // //////////////////////////////////////////////////////////////////// using System; using System.Collections; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private ArrayList arrRecoMenu; private InkControl2 inkCtl; private RecognizerContext recoCtxt; private StrokesEventHandler evtInkAdded; private StrokesEventHandler evtInkDeleted; private static Recognizers recognizers = new Recognizers(); // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed by analyzing // the number of items in the Recognizers collection if (frmMain.recognizers.Count > 0) { Application.Run(new frmMain()); } else { // None are, so display error message and exit MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } // Main form setup public frmMain() { SuspendLayout(); // Create the main menu Menu = new MainMenu(); MenuItem miFile = new MenuItem("&File"); Menu.MenuItems.Add(miFile); MenuItem miExit = new MenuItem("E&xit"); miExit.Click += new EventHandler(miExit_Click); Menu.MenuItems[0].MenuItems.Add(miExit); MenuItem miRecognizers = new MenuItem("&Recognizers"); Menu.MenuItems.Add(miRecognizers); // Create a menu full of the installed recognizers arrRecoMenu = new ArrayList(); foreach (Recognizer recognizer in frmMain.recognizers) { MenuItem miRecoInst = new MenuItem(recognizer.Name); miRecoInst.Click += new EventHandler(miRecoInst_Click); miRecoInst.RadioCheck = true; Menu.MenuItems[1].MenuItems.Add(miRecoInst); arrRecoMenu.Add(miRecoInst); } MenuItem miSeparator = new MenuItem("-"); Menu.MenuItems[1].MenuItems.Add(miSeparator); MenuItem miRecognize = new MenuItem("Recognize!"); miRecognize.Click += new EventHandler(miRecognize_Click); Menu.MenuItems[1].MenuItems.Add(miRecognize); // Create and place all of our controls inkCtl = new InkControl2(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); // Configure the form itself ClientSize = new Size(368, 232); Controls.AddRange(new Control[] { inkCtl }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "AdvancedReco"; ResumeLayout(false); // Start off using the default recognizer SwitchToRecognizer(frmMain.recognizers.GetDefaultRecognizer()); // Get notification of ink being added and deleted so we'll know // when to add and remove ink from the reco context inkCtl.InkOverlay.Ink.InkAdded += new StrokesEventHandler( inkCtl_InkAdded); inkCtl.InkOverlay.Ink.InkDeleted += new StrokesEventHandler( inkCtl_InkDeleted); // Create event handlers so we can be called back on the correct // thread evtInkAdded = new StrokesEventHandler(inkCtl_InkAdded_Apt); evtInkDeleted = new StrokesEventHandler(inkCtl_InkDeleted_Apt); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Sets up recognition using the given Recognizer instance private void SwitchToRecognizer(Recognizer recognizer) { // Create a recognizer context and set strokes on it recoCtxt = recognizer.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; // Update the menu items with the new choice foreach (MenuItem miReco in arrRecoMenu) { miReco.Checked = miReco.Text.Equals(recognizer.Name); } } // Given a name, finds the Recognizer instance private Recognizer FindRecognizer(string strName) { foreach (Recognizer recognizer in frmMain.recognizers) { if (recognizer.Name.Equals(strName)) { return recognizer; } } return null; } // Handle new ink being added private void inkCtl_InkAdded(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkAdded, new object[] { sender, e }); } private void inkCtl_InkAdded_Apt(object sender, StrokesEventArgs e) { // Update the strokes to recognize recoCtxt.Strokes.Add( inkCtl.InkOverlay.Ink.CreateStrokes(e.StrokeIds)); } // Handle ink getting deleted private void inkCtl_InkDeleted(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkDeleted, new object[] { sender, e }); } private void inkCtl_InkDeleted_Apt(object sender, StrokesEventArgs e) { // Update the strokes to recognize - remove any that have been // deleted Strokes strksRemove = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke strk in recoCtxt.Strokes) { if (strk.Deleted) { strksRemove.Add(strk); } } recoCtxt.Strokes.Remove(strksRemove); } // Handle "Exit" menu item private void miExit_Click(object sender, EventArgs e) { Application.Exit(); } // Handle a recognizer menu item private void miRecoInst_Click(object sender, EventArgs e) { MenuItem miNewReco = sender as MenuItem; if (miNewReco != null) { SwitchToRecognizer(FindRecognizer(miNewReco.Text)); } } // Handle Recognize! menu item private void miRecognize_Click(object sender, EventArgs e) { if (recoCtxt.Strokes.Count > 0) { // No more strokes will be input at this point recoCtxt.EndInkInput(); // Obtain the recognition results RecognitionStatus recoStatus; RecognitionResult recoResult = recoCtxt.Recognize(out recoStatus); if (recoStatus == RecognitionStatus.NoError) { // Show the results MessageBox.Show(this, recoResult.TopString, "Results"); } else { // Problem; display error message MessageBox.Show(this, recoStatus.ToString(), "Error"); } // Reset the strokes to recognize for next time, "undoing" // what EndInkInput has done. recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; } } } Notice the event handlers for the InkAdded and InkDeleted events, and recall from Chapter 5 that they can be fired on a thread other than the application s main UI thread of execution. Therefore, to keep data access properly protected, the Control class s Invoke method is used to manually retrigger the event on the application s main thread.
The InkAdded handler simply adds the new strokes to the reco context s Strokes collection, and the InkDeleted handler removes the deleted strokes from the reco context. To remove the strokes from the reco context, the application computes a Strokes collection of the strokes that have their Deleted property set and then removes those strokes from the reco context s Strokes collection:
Strokes strksRemove = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke strk in recoCtxt.Strokes) { if (strk.Deleted) { strksRemove.Add(strk); } } recoCtxt.Strokes.Remove(strksRemove); The Recognize! menu item handler looks rather similar to the Intermedi ateReco application s version; the key differences here are that no recognizer instance is obtained nor is a recognizer context created because that has already occurred at application startup. In addition, once the recognized results have been displayed, the reco context s Strokes collection is reassigned. The reason for this reassignment is that we want to keep the reco context around for further use, and it needs to be reset for more strokes to be added to or removed from its collection because EndInkInput was called.
Contrast the performance of this application with the previous sample write a couple of sentences, wait a few seconds, and choose the Recognize! command. The IntermediateReco sample will pause for a few seconds and then display the message box, whereas this sample should show the results in about half the time. This is partial recognition at work!
After dismissing the message box containing the results, quickly choose the Recognize! command again. The results take longer to appear because the recognition operation is reset by reassigning the reco context s Strokes collection after the call to the Recognize method. The application does this so that it can work regardless of whether the recognizer being used supports partial recognition. Although all the text recognizers in the Tablet PC Platform support partial recognition, future recognizers provided by Microsoft or third parties might not. This could be important for you to consider if you plan on your application supporting arbitrary recognizers. If you are certain that the recognizers your application uses support partial recognition, you can improve efficiency by not calling the RecognizerContext.EndInkInput method before RecognizerContext.Recognize and not reassigning the reco context s Strokes collection to avoid resetting the reco operation.
Now try the following modification to the sample in the Recognize! menu item handler:
if (recoCtxt.Strokes.Count > 0) { // No more strokes will be input at this point //REMOVED: recoCtxt.EndInkInput(); // Obtain the recognition results RecognitionStatus recoStatus; RecognitionResult recoResult = recoCtxt.Recognize(out recoStatus); if (recoStatus == RecognitionStatus.NoError) { // Show the results MessageBox.Show(this, recoResult.TopString, "Results"); } else { // Problem; display error message MessageBox.Show(this, recoStatus.ToString(), "Error"); } // Reset the strokes to recognize for next time, "undoing" // what EndInkInput has done. //REMOVED: recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; } This modification is targeted toward recognizers that support partial recognition. When using a recognizer that supports partial recognition, the results will be returned quickly no matter how soon subsequent recognition operations are triggered. If the recognizer used does not support partial recognition, you will encounter an error when the Recognize method is called.
NOTE
The Recognize method can be used to determine whether a recognizer supports partial recognition or not, given that a Recognizer object, a reco context, an Ink object, and a Stroke object can be programmatically created and a recognition attempt can be made on the ink without calling EndInkInput. If an error occurs from calling the reco context s Recognize method, the recognizer does not support partial recognition.
Using the Microsoft Gesture Recognizer
The Microsoft Gesture Recognizer cannot be used from the recognition classes because there is currently no way to specify what application gestures are desired, as there is for the InkCollector and InkOverlay classes (the SetGestureStatus method). Hence, if your application requires gesture recognition, you will have to use either the InkCollector or InkOverlay class and its Gesture event. However, there is nothing preventing a custom-developed gesture recognizer from being used with the recognition API; the issue being discussed here applies only to the Microsoft Gesture Recognizer.Getting Results III: The Final Chapter (a.k.a. Asynchronous Recognition)
Obtaining recognition results by the Recognize method can be time-consuming, even when the recognizer being used supports partial recognition and the reco context s Strokes collection is kept up-to-date at all times. For some application features it is useful to receive notification that recognition results have been computed rather than block while waiting for Recognize to return. Asynchronous recognition is the answer to this problem instead of waiting for a method to return while the recognition results are being computed, the method to request recognition results returns immediately and the results are provided later from an event.
To perform asynchronous recognition, either of the RecognizerContext class s methods, BackgroundRecognize or BackgroundRecognizeWithAlternates, is called. Whichever one is called returns immediately, and when recognition results are computed then either the Recognition or RecognitionWithAlternates event is fired. The difference between the two methods is how much information is computed and supplied in their respective event: the BackgroundRecognize method provides only the top string from the Recognition event, whereas BackgroundRecognizeWithAlternates provides a full RecognitionResult instance to the RecognitionWithAlternates event. The methods are rather like asynchronous versions of Strokes.ToString and RecognizerContext.Recognize, respectively.
If the recognizer being used supports partial recognition and the recognition context is kept up-to-date with ink, it is actually a little more efficient to perform recognition asynchronously than it is to do so synchronously. The utility of requesting recognition results in an asynchronous manner is that the requesting thread is not blocked while waiting for the computation, allowing for further user input.
Sample Application AsyncReco
This sample application, shown in Figure 7-2, illustrates how asynchronous recognition can be performed. As strokes are added to and removed from an InkControl s Ink object, the recognition results are updated in real time in an edit box effectively forming a log of results.
Figure 7-2. The AsyncReco application displays recognition results as they re calculated.
AsyncReco.cs
//////////////////////////////////////////////////////////////////// // // AsyncReco.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program demonstrates using asychronous recognition. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private InkControl2 inkCtl; private TextBox txtResults; private RecognizerContext recoCtxt; private StrokesEventHandler evtInkAdded; private StrokesEventHandler evtInkDeleted; private RecognizerContextRecognitionEventHandler evtRecognition; // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed if ((new Recognizers()).Count > 0) { Application.Run(new frmMain()); } else { // None are, so display error message and exit MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } // Main form setup public frmMain() { SuspendLayout(); // Create and place all of our controls inkCtl = new InkControl2(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); txtResults = new TextBox(); txtResults.Location = new Point(8, 232); txtResults.Multiline = true; txtResults.ReadOnly = true; txtResults.ScrollBars = ScrollBars.Vertical; txtResults.Size = new Size(352, 72); // Configure the form itself ClientSize = new Size(368, 312); Controls.AddRange(new Control[] { inkCtl, txtResults}); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "AsyncReco"; ResumeLayout(false); // Get the default recognizer Recognizer reco = (new Recognizers()).GetDefaultRecognizer(); // Create a recognizer context recoCtxt = reco.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.CreateStrokes(); // Get notification of ink being added and deleted so we'll know // when to stop and start background recognition inkCtl.InkOverlay.Ink.InkAdded += new StrokesEventHandler( inkCtl_InkAdded); inkCtl.InkOverlay.Ink.InkDeleted += new StrokesEventHandler( inkCtl_InkDeleted); // Get notification of recognition occurring to display the // current results recoCtxt.Recognition += new RecognizerContextRecognitionEventHandler( recoCtxt_Recognition); // Create event handlers so we can be called back on the correct // thread evtInkAdded = new StrokesEventHandler(inkCtl_InkAdded_Apt); evtInkDeleted = new StrokesEventHandler(inkCtl_InkDeleted_Apt); evtRecognition = new RecognizerContextRecognitionEventHandler( recoCtxt_Recognition_Apt); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Handle new ink being added private void inkCtl_InkAdded(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkAdded, new object[] { sender, e }); } private void inkCtl_InkAdded_Apt(object sender, StrokesEventArgs e) { // Stop any current background recognition recoCtxt.StopBackgroundRecognition(); // Update the strokes to recognize recoCtxt.Strokes.Add( inkCtl.InkOverlay.Ink.CreateStrokes(e.StrokeIds)); // No more strokes will be input at this point recoCtxt.EndInkInput(); // Restart background recognition recoCtxt.BackgroundRecognize(); } // Handle ink being deleted private void inkCtl_InkDeleted(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkDeleted, new object[] { sender, e }); } private void inkCtl_InkDeleted_Apt(object sender, StrokesEventArgs e) { // Stop any current background recognition recoCtxt.StopBackgroundRecognition(); // Figure out which strokes have been deleted Strokes strksRemove = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke strk in recoCtxt.Strokes) { if (strk.Deleted) { strksRemove.Add(strk); } } // Update the strokes to recognize recoCtxt.Strokes.Remove(strksRemove); // Check if we can restart background recognition if (recoCtxt.Strokes.Count > 0) { // No more strokes will be input at this point recoCtxt.EndInkInput(); // Restart background recognition recoCtxt.BackgroundRecognize(); } else { // No strokes left, so empty results txtResults.Text = ""; } } // Handle recognition results being computed private void recoCtxt_Recognition( object sender, RecognizerContextRecognitionEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtRecognition, new object[] { sender, e }); } private void recoCtxt_Recognition_Apt( object sender, RecognizerContextRecognitionEventArgs e) { if (e.RecognitionStatus == RecognitionStatus.NoError) { // Display the result txtResults.AppendText(e.Text + "\r\n"); } else { // Display the recognition status txtResults.AppendText("Problem: RecoStatus="); txtResults.AppendText(e.RecognitionStatus.ToString()); txtResults.AppendText("\r\n"); } // Reset the strokes to recognize for next time, "undoing" // what EndInkInput has done. recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; } } The first part of the application is similar to what we ve seen in the IntermediateReco and AdvancedReco samples at least one recognizer is confirmed to be installed for the application to run. After creating a form with an InkControl and a TextBox, a RecognizerContext is created from the default recognizer. The Recognition event has a handler installed for it in the same fashion as InkAdded and InkDeleted do because the Recognition event can be fired on a thread other than the main application thread.
The handling of the InkAdded and InkDeleted events is similar to, but not quite the same as, the other samples that keep the reco context up-to-date with ink. Let s take a look at the logic found in the InkAdded event handler:
// Stop any current background recognition recoCtxt.StopBackgroundRecognition(); // Update the strokes to recognize recoCtxt.Strokes.Add( inkCtl.InkOverlay.Ink.CreateStrokes(e.StrokeIds)); // No more strokes will be input at this point recoCtxt.EndInkInput(); // Restart background recognition recoCtxt.BackgroundRecognize();
The RecognizerContext.StopBackgroundRecognition method results in the cancellation of any in-progress asynchronous recognition operation. We ll see shortly why we d want to call that method at the top of the event handler. The modification of the reco context s Strokes collection is straightforward; it simply adds the new ink strokes to the collection. The EndInkInput method is then called because a recognition operation is about to be requested namely, the BackgroundRecognize method. This is why the call to StopBackgroundRecognition is needed at the top of the function. If ink strokes are added in rapid succession, it s quite possible that an asynchronous recognition operation is still in progress when the reco context s Strokes collection is about to be modified with the new ink. To avoid this run-time error condition we therefore cancel any recognition operation by calling StopBackgroundRecognition before touching the reco context s Strokes collection.
The Recognition event handler receives a RecognizerContextRecognitionEventArgs object containing the result of asynchronous recognition. The event arguments provide no other information other than the recognition status and an optional object reference passed into BackgroundRecognize. To get a RecognitionResult instance from an asynchronous recognition operation, we need to call BackgroundRecognizeWithAlternates. In this case, the RecognitionWithAlternate event s RecognizerContextRecognitionWithAlternatesEventArgs class contains a property named Result that holds the computed RecognitionResult.
Working with Recognition Results
Now that we know how to get hold of a RecognitionResult instance, what exactly can it be used for? As mentioned earlier, we know the class encapsulates recognized result data such as text, alternative results, and confidence level. This section discusses how to work with that information to deliver the best possible user experience from an ink-enabled application.
Recognition result data is compiled from a complex data structure containing all the possible results for ink data that has been recognized. Known as a lattice, it is a matrix-like structure produced by a recognizer when a recognition operation has been completed. You can imagine it to be something like a large table whose columns define the segmentation of ink and whose rows define the possible results. The data in a lattice comprises cells containing recognized text, confidence level, ink segmentation, line information, and stroke association with the recognized text. Alternates define a path through cells of the lattice from adjacent columns. Don t fret if this isn t making much sense at this point just keep in mind that alternates represent the recognition results for consecutive segments of ink.
A RecognitionResult instance has what s known as a top alternate; this is the path through the lattice that has the highest confidence. We saw from the IntermediateReco and AdvancedReco samples that the RecognitionResult class s TopString property returns the text string with highest confidence. Similarly, the TopConfidence property indicates the confidence level of the top alternate. Because the top alternate can be (and usually is) made up of multiple segments, each containing a confidence value, the TopConfidence property is actually an average of values.
The TopAlternate property of the RecognitionResult class provides the top alternate for the result, returning an instance of the RecognitionAlternate class. To access other alternates from a RecognitionResult instance, the GetAlternatesFromSelection method returns a collection of RecognitionAlternate instances by way of the RecognitionAlternates class. The method can optionally accept a start position and length of text from the result string in order to drill down into the results. The CorrectionUI sample later in this chapter makes use of this method to display the alternative results for an arbitrary selection.
The RecognitionAlternate Class
Once a RecognitionAlternate instance has been obtained from a RecognitionResult, we can access the data the recognizer has computed as well as further explore the recognition results. Table 7-8 lists some common properties found on a RecognitionAlternate instance.
Value | Type | Description |
Ascender | Line | The ascender line computed for the strokes drawn; the upper boundary of the characters |
Baseline | Line | The baseline computed for the strokes drawn; equivalent to the underline of the characters |
Confidence | RecognitionConfidence | The confidence level for the entire alternate; an average of the confidence values of the referenced lattice cells |
Descender | Line | The descender line computed for the strokes drawn; the lower boundary of the characters |
LineNumber | Int | The line number the strokes were computed on; 0 = first, 1 = second, etc. |
Midline | Line | The midline computed for the strokes drawn; lies between the baseline and ascender |
Strokes | Strokes | The strokes associated with this alternate |
The ConfidenceAlternates property of the RecognitionAlternate class returns a RecognitionAlternates collection, effectively subdividing the alternate into consecutive pieces that share the same confidence level. For example, if the segments in the sentence, I m an operator of my pocket calculator, had confidence levels of high, high, poor, intermediate, intermediate, intermediate, and high respectively, four alternates corresponding to I m an, operator, of my pocket, and calculator would be returned from the ConfidenceAlternates property.
Similarly, the LineAlternates property subdivides an alternate into pieces that share the same line. Ink can be entered on multiple lines, of course, so this property is handy in order to find out what strokes lie on what line.
Querying Line-Based Properties
Querying the Ascender, Baseline, Descender, LineNumber, and Midline properties will throw a run-time exception if the alternate spans multiple lines of ink. This is because the properties are line-based, meaning their value applies to a specific line. To ensure it is safe to query them, you can either check that the count of alternates returned from the LineAlternates property is 1, or iterate over each alternate returned from the LineAlternates property, querying each one.It can be useful to match ink strokes to character positions in the recognized text string and vice versa; correction UI and word-based selection, for instance, benefit greatly from this functionality. The GetStrokesFromTextRange, GetTextRangeFromStrokes, and GetStrokesFromStrokeRanges methods are used to accomplish this. The GetStrokesFromTextRange method computes the Strokes collection that references the ink strokes forming the segments specified by the character position and length in the recognized text. Likewise, the GetTextRangeFromStrokes method computes the character position and length in the recognized text that is formed by the specified strokes. A useful feature of these functions is that they automatically snap to a segment boundary, meaning the data provided to the methods doesn t have to contain all strokes or characters forming a word.
The GetStrokesFromStrokeRanges method is a combination of GetText RangeFromStrokes and GetStrokesFromTextRange. Given one or more Strokes collections, this method returns the Strokes collection for those segments that are spanned. You can get the same value by calling GetTextRangeFromStrokes and then passing the return value from that method to GetStrokesFromText Range. GetStrokesFromStrokeRanges is a handy method for performing word selection, as we re about to see.
Sample Application WordSelect
The WordSelect sample application shows how we can make use of recognition results to perform word-based selection. If you compare the selection behavior of Windows Journal and the InkOverlay class, you ll notice that tapping on a word in Windows Journal results in the entire word being selected, whereas InkOverlay selects only the single stroke hit by the tap.
By taking advantage of the GetStrokesFromStrokeRanges method of the RecognitionAlternate class, we can easily change InkOverlay s behavior to select the entire word.
WordSelect.cs
//////////////////////////////////////////////////////////////////// // // WordSelect.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program demonstrates how to select full words based on // recognition results. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private InkControl2 inkCtl; private TextBox txtResults; private RecognizerContext recoCtxt; private RecognitionResult recoResult; private StrokesEventHandler evtInkAdded; private StrokesEventHandler evtInkDeleted; private RecognizerContextRecognitionWithAlternatesEventHandler evtRecognitionWithAlternates; // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed if ((new Recognizers()).Count > 0) { Application.Run(new frmMain()); } else { // None are, so display error message and exit MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } // Main form setup public frmMain() { SuspendLayout(); // Create and place all of our controls inkCtl = new InkControl2(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); txtResults = new TextBox(); txtResults.HideSelection = false; txtResults.Location = new Point(8, 232); txtResults.ReadOnly = true; txtResults.ScrollBars = ScrollBars.Vertical; txtResults.Size = new Size(352, 20); // Configure the form itself ClientSize = new Size(368, 260); Controls.AddRange(new Control[] { inkCtl, txtResults}); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "WordSelect"; ResumeLayout(false); // Get the default recognizer Recognizer reco = (new Recognizers()).GetDefaultRecognizer(); // Create a recognizer context recoCtxt = reco.CreateRecognizerContext(); recoCtxt.RecognitionFlags = RecognitionModes.TopInkBreaksOnly; recoCtxt.Strokes = inkCtl.InkOverlay.Ink.CreateStrokes(); // Get notification of ink being added and deleted so we'll know // when to stop and start background recognition inkCtl.InkOverlay.Ink.InkAdded += new StrokesEventHandler( inkCtl_InkAdded); inkCtl.InkOverlay.Ink.InkDeleted += new StrokesEventHandler( inkCtl_InkDeleted); // Get notification of selection changing so we can tweak the // result if needed inkCtl.InkOverlay.SelectionChanging += new InkOverlaySelectionChangingEventHandler( inkCtl_SelectionChanging); // Get notification of recognition occurring to display the // current results recoCtxt.RecognitionWithAlternates += new RecognizerContextRecognitionWithAlternatesEventHandler( recoCtxt_RecognitionWithAlternates); // Create event handlers so we can be called back on the correct // thread evtInkAdded = new StrokesEventHandler(inkCtl_InkAdded_Apt); evtInkDeleted = new StrokesEventHandler(inkCtl_InkDeleted_Apt); evtRecognitionWithAlternates = new RecognizerContextRecognitionWithAlternatesEventHandler( recoCtxt_RecognitionWithAlternates_Apt); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Handle new ink being added private void inkCtl_InkAdded(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkAdded, new object[] { sender, e }); } private void inkCtl_InkAdded_Apt(object sender, StrokesEventArgs e) { // Stop any current background recognition recoCtxt.StopBackgroundRecognition(); // Update the strokes to recognize recoCtxt.Strokes.Add( inkCtl.InkOverlay.Ink.CreateStrokes(e.StrokeIds)); // No more strokes will be input at this point recoCtxt.EndInkInput(); // Restart background recognition recoCtxt.BackgroundRecognizeWithAlternates(); } // Handle ink getting deleted private void inkCtl_InkDeleted(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkDeleted, new object[] { sender, e }); } private void inkCtl_InkDeleted_Apt(object sender, StrokesEventArgs e) { // Stop any current background recognition recoCtxt.StopBackgroundRecognition(); // Figure out which strokes have been deleted Strokes strksRemove = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke strk in recoCtxt.Strokes) { if (strk.Deleted) { strksRemove.Add(strk); } } // Update the strokes to recognize recoCtxt.Strokes.Remove(strksRemove); // Check if we can restart background recognition if (recoCtxt.Strokes.Count > 0) { // No more strokes will be input at this point recoCtxt.EndInkInput(); // Restart background recognition recoCtxt.BackgroundRecognizeWithAlternates(); } else { // No strokes left, so empty results txtResults.Text = ""; } } // Handle ink getting selected private void inkCtl_SelectionChanging( object sender, InkOverlaySelectionChangingEventArgs e) { // Stop any current background recognition to make sure we // have stable results recoCtxt.StopBackgroundRecognition(); if ((e.NewSelection.Count > 0) && (recoResult != null)) { // Select the strokes comprising a full word using the // GetStrokesFromStrokeRanges method // Just using this method would cause an entire range to // always be selected, breaking expected lasso behavior //e.NewSelection.Add( // recoResult.TopAlternate.GetStrokesFromStrokeRanges( // e.NewSelection)); // Select the full words that contain the selected stroke(s) Strokes strksToSelect = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke s in e.NewSelection) { if (strksToSelect.Contains(s)) { continue; } Strokes strksStroke = inkCtl.InkOverlay.Ink.CreateStrokes( new int[] { s.Id } ); Strokes strksWord = recoResult.TopAlternate.GetStrokesFromStrokeRanges( strksStroke); strksToSelect.Add(strksWord); } e.NewSelection.Clear(); e.NewSelection.Add(strksToSelect); // Select the text in the results textbox corresponding // to the new selection - since the textbox doesn't support // multi-ranged selection, we'll have to select the entire // range of words defined by the new selection int nStart = 0, nLength = 0; recoResult.TopAlternate.GetTextRangeFromStrokes( e.NewSelection, ref nStart, ref nLength); txtResults.Select(nStart, nLength); } else { // Clear the selection in the textbox txtResults.Select(0, 0); } } // Handle recognition results being computed private void recoCtxt_RecognitionWithAlternates(object sender, RecognizerContextRecognitionWithAlternatesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke( evtRecognitionWithAlternates, new object[] { sender, e }); } private void recoCtxt_RecognitionWithAlternates_Apt(object sender, RecognizerContextRecognitionWithAlternatesEventArgs e) { if (e.RecognitionStatus == RecognitionStatus.NoError) { // Check to see if these reco results are needed - if there // are no strokes in the recognition context, then these // results are stale if (recoCtxt.Strokes.Count > 0) { // Store the result for later recoResult = e.Result; // Display the result txtResults.Text = e.Result.TopString; } else { // Clear the displayed results txtResults.Text = ""; } } else { // Clear the stored result recoResult = null; // Display the recognition status txtResults.Text = "Problem: RecoStatus=" + e.RecognitionStatus.ToString(); } // Reset the strokes to recognize for next time, "undoing" // what EndInkInput has done. recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; } } Similar to the AsyncReco sample we saw earlier, background recognition is used to display the current recognition result in real time. Instead of using BackgroundRecognize, however, we use BackgroundRecognizeWithAlternates to obtain a RecognitionResult instance. This recognition result is saved each time the RecognitionWithAlternates event fires so that the application always has up-to-date results from which to query.
When stroke selection occurs, the strokes being selected are used as the argument to the saved recognition result s top alternate s GetStrokesFromStroke Ranges method. The strokes constituting both the words and the text in the results textbox are then selected. Because only the top alternate is used from the recognition result, we need only one set of segmentation, so for efficiency we can specify the RecognitionModes.TopInkBreaksOnly flag to the recognition context.
Word selection is achieved with the following code in the SelectionChanging event handler:
// Select the full words that contain the selected stroke(s) Strokes strksToSelect = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke s in e.NewSelection) { if (strksToSelect.Contains(s)) { continue; } Strokes strksStroke = inkCtl.InkOverlay.Ink.CreateStrokes( new int[] { s.Id } ); Strokes strksWord = recoResult.TopAlternate.GetStrokesFromStrokeRanges( strksStroke); strksToSelect.Add(strksWord); } e.NewSelection.Clear(); e.NewSelection.Add(strksToSelect); Each stroke to be selected is given individually to GetStrokesFromStroke Ranges, which effectively performs a word-by-word calculation. The resulting Strokes collections are then added to one master Strokes collection, and finally the InkOverlaySelectionChangingEventArgs s NewSelection property is set to this master collection.
You might think that instead of this sequence, GetStrokesFromStroke Ranges could just be called with the entire contents of the NewSelection property. However, this would result in incorrect lasso selection behavior when multiple words were selected. For example, if the user wrote two lines of text such as My name is and Rico Suav and then tried to lasso select only the words My and Rico, the words My name is Rico would get selected because GetStrokesFromStrokeRanges returns not only the strokes for the two words, but for all the words in between.
Sample Application CorrectionUI
This next sample application shows how a user can edit a RecognitionResult instance from a correction UI. The sample implements a dialog enabling the user to choose alternate recognition results and then copy the recognized text to the clipboard. For convenience, the implementation of the dialog is found in the BuildingTabletApps helper library introduced in Chapter 4. Figure 7-3 shows what the CorrectionUI application looks like.
Figure 7-3. The CorrectionUI sample application provides the user with the ability to alter the top alternate result.
CorrectionUI.cs
//////////////////////////////////////////////////////////////////// // // CorrectionUI.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This program shows how to display correction UI for a "Copy As // Text" menu item. // //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; using MSPress.BuildingTabletApps; public class frmMain : Form { private InkControl2 inkCtl; private RecognizerContext recoCtxt; private StrokesEventHandler evtInkAdded; private StrokesEventHandler evtInkDeleted; // Entry point of the program [STAThread] static void Main() { // Check to see if any recognizers are installed by analyzing // the number of items in the Recognizers collection Recognizers recognizers = new Recognizers(); if (recognizers.Count > 0) { Application.Run(new frmMain()); } else { // None are, so display error message and exit MessageBox.Show("No recognizers are installed!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } // Main form setup public frmMain() { SuspendLayout(); // Create the main menu Menu = new MainMenu(); MenuItem miFile = new MenuItem("&File"); Menu.MenuItems.Add(miFile); MenuItem miExit = new MenuItem("E&xit"); miExit.Click += new EventHandler(miExit_Click); Menu.MenuItems[0].MenuItems.Add(miExit); MenuItem miEdit = new MenuItem("&Edit"); Menu.MenuItems.Add(miEdit); MenuItem miCopyAsText = new MenuItem("&Copy As Text"); miCopyAsText.Click += new EventHandler(miCopyAsText_Click); Menu.MenuItems[1].MenuItems.Add(miCopyAsText); // Create and place all of our controls inkCtl = new InkControl2(); inkCtl.Location = new Point(8, 8); inkCtl.Size = new Size(352, 216); // Configure the form itself ClientSize = new Size(368, 232); Controls.AddRange(new Control[] { inkCtl }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "CorrectionUI"; ResumeLayout(false); // Get the default recognizer Recognizer reco = (new Recognizers()).GetDefaultRecognizer(); // Create a recognizer context and hook up to recognition event recoCtxt = reco.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.CreateStrokes(); // Get notification of ink being added and deleted so we'll know // when to add and remove ink from the reco context inkCtl.InkOverlay.Ink.InkAdded += new StrokesEventHandler( inkCtl_InkAdded); inkCtl.InkOverlay.Ink.InkDeleted += new StrokesEventHandler( inkCtl_InkDeleted); // Create event handlers so we can be called back on the correct // thread evtInkAdded = new StrokesEventHandler(inkCtl_InkAdded_Apt); evtInkDeleted = new StrokesEventHandler(inkCtl_InkDeleted_Apt); // We're now set to go, so turn on tablet input inkCtl.InkOverlay.Enabled = true; } // Handle new ink being added private void inkCtl_InkAdded(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkAdded, new object[] { sender, e }); } private void inkCtl_InkAdded_Apt(object sender, StrokesEventArgs e) { // Update the strokes to recognize recoCtxt.Strokes.Add( inkCtl.InkOverlay.Ink.CreateStrokes(e.StrokeIds)); } // Handle ink getting deleted private void inkCtl_InkDeleted(object sender, StrokesEventArgs e) { // Make sure the event fires on the correct thread this.Invoke(evtInkDeleted, new object[] { sender, e }); } private void inkCtl_InkDeleted_Apt(object sender, StrokesEventArgs e) { // Update the strokes to recognize - remove any that have been // deleted Strokes strksRemove = inkCtl.InkOverlay.Ink.CreateStrokes(); foreach (Stroke strk in recoCtxt.Strokes) { if (strk.Deleted) { strksRemove.Add(strk); } } recoCtxt.Strokes.Remove(strksRemove); } // Handle "Exit" menu item private void miExit_Click(object sender, EventArgs e) { Application.Exit(); } // Handle "Copy As Text" menu item private void miCopyAsText_Click(object sender, EventArgs e) { if (recoCtxt.Strokes.Count > 0) { // No more strokes will be input at this point recoCtxt.EndInkInput(); // Obtain the recognition results RecognitionStatus recoStatus; RecognitionResult recoResult = recoCtxt.Recognize(out recoStatus); if (recoStatus == RecognitionStatus.NoError) { // Show correction UI dialog CopyAsTextDlg dlg = new CopyAsTextDlg(); dlg.RecognitionResult = recoResult; dlg.ShowDialog(this); } else { // Problem; display error message MessageBox.Show(this, "Error: " + recoStatus); } // Reset the strokes to recognize for next time, "undoing" // what EndInkInput has done. recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; } } } This first file is the main program that calls the dialog implementation found in the BuildingTabletApps library. It is similar to the AdvancedReco application in that it keeps the reco context up-to-date with the ink to recognize the Copy As Text menu item handler obtains the reco results synchronously by a call to Recognize. The RecognitionResult value returned is then provided as the argument to the CopyAsTextDlg class, which performs all the neat editing functionality.
The implementation of CopyAsTextDlg is presented here:
CopyAsTextDlg.cs
//////////////////////////////////////////////////////////////////// // // CopyAsTextDlg.cs // // (c) 2002 Microsoft Press // by Rob Jarrett // // This class implements a dialog which provides correction and // clipboard copying of reco results via a RecognitionResults object. // //////////////////////////////////////////////////////////////////// using System; using System.Collections; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; namespace MSPress.BuildingTabletApps { public class CopyAsTextDlg : Form { private TextBox txtResults; private ListBox lbAlternates; private InkInputPanel pnlInk; private Button btnCopy; private Button btnClose; private RecognitionResult recoResult; private Strokes strksDisplay; // Dialog setup public CopyAsTextDlg() { SuspendLayout(); // Create and place all of our controls txtResults = new TextBox(); txtResults.HideSelection = false; txtResults.Location = new Point(8, 8); txtResults.Multiline = true; txtResults.ReadOnly = true; txtResults.ScrollBars = ScrollBars.Vertical; txtResults.Size = new Size(384, 96); lbAlternates = new ListBox(); lbAlternates.Location = new Point(8, 112); lbAlternates.Size = new Size(188, 96); pnlInk = new InkInputPanel(); pnlInk.BackColor = Color.White; pnlInk.BorderStyle = BorderStyle.Fixed3D; pnlInk.Location = new Point(204, 112); pnlInk.Size = new Size(188, 96); btnCopy = new Button(); btnCopy.DialogResult = DialogResult.OK; btnCopy.Location = new Point(264, 226); btnCopy.Size = new Size(60, 20); btnCopy.Text = "Copy"; btnClose = new Button(); btnClose.DialogResult = DialogResult.OK; btnClose.Location = new Point(332, 226); btnClose.Size = new Size(60, 20); btnClose.Text = "Close"; // Configure the form itself AcceptButton = btnClose; CancelButton = btnClose; ClientSize = new Size(400, 256); Controls.AddRange(new Control[] { txtResults, lbAlternates, pnlInk, btnCopy, btnClose}); FormBorderStyle = FormBorderStyle.FixedDialog; MinimizeBox = false; MaximizeBox = false; Text = "Copy As Text"; ResumeLayout(false); // Get notification of user input occuring in the "Results" // TextBox so we can update the "Alternates" ListBox txtResults.KeyUp += new KeyEventHandler(txtResults_KeyUp); txtResults.MouseUp += new MouseEventHandler(txtResults_MouseUp); // Get notification of an alternate getting chosen so we // can alter the result lbAlternates.SelectedIndexChanged += new EventHandler(lbAlternates_SelectedIndexChanged); // Paint the preview of the strokes making up the selection pnlInk.Paint += new PaintEventHandler(pnlInk_Paint); // Perform copying of the text btnCopy.Click += new EventHandler(btnCopy_Click); } // Get or set the RecognitionResult object being used for the // correction public RecognitionResult RecognitionResult { get { return recoResult; } set { recoResult = value; if (recoResult != null) { // Display the current result txtResults.Text = recoResult.TopString; txtResults.SelectionStart = 0; txtResults.SelectionLength = 0; SelectionChanged(); } } } private void FillListbox(RecognitionAlternates recoAlternates) { // Clear out any current items in the listbox lbAlternates.BeginUpdate(); lbAlternates.Items.Clear(); // Fill up the listbox with each alternate foreach (RecognitionAlternate recoAlternate in recoAlternates) { lbAlternates.Items.Add(recoAlternate); } lbAlternates.EndUpdate(); } // The selection in the results textbox has changed, so figure // out what word(s) were selected, "snap" the selection to them // if needed, fill the listbox, and update the preview with the // ink being used private void SelectionChanged() { // Figure out the best text range in the "Results" TextBox // to use in obtaining the alternates int nStart = txtResults.SelectionStart; int nLength = txtResults.SelectionLength; if (nLength == 0) { if (nStart < txtResults.Text.Length && txtResults.Text.Length > 0) { nLength = 1; } else if (nStart > 0) { nStart--; nLength = 1; } } // If we still have no selection, no need to do anything if (nLength == 0) { return; } // Select the strokes that represent the selection in the // "Results" Textbox strksDisplay = recoResult.TopAlternate.GetStrokesFromTextRange( ref nStart, ref nLength); txtResults.Select(nStart, nLength); pnlInk.Invalidate(); // Show the list of alternates for the selection FillListbox( recoResult.GetAlternatesFromSelection(nStart, nLength)); } // Handle a key being released private void txtResults_KeyUp(object sender, KeyEventArgs e) { // Selection probably changed, so update the alternates listbox SelectionChanged(); } // Handle a mouse button being released private void txtResults_MouseUp(object sender, MouseEventArgs e) { // Selection probably changed, so update the alternates listbox SelectionChanged(); } // Handle an alternate getting selected private void lbAlternates_SelectedIndexChanged( object sender, EventArgs e) { // Get the alternate that was selected RecognitionAlternate recoAlternate = (RecognitionAlternate)lbAlternates.SelectedItem; // Adjust the current reco results with it recoResult.ModifyTopAlternate(recoAlternate); // Update selected text int nStart = txtResults.SelectionStart; int nLength = txtResults.SelectionLength; recoResult.TopAlternate.GetTextRangeFromStrokes( strksDisplay, ref nStart, ref nLength); // Update the results edit box txtResults.Text = recoResult.TopString; txtResults.Select(nStart, nLength); // Show the list of alternates for the selection FillListbox( recoResult.GetAlternatesFromSelection(nStart, nLength)); } // Handle painting the preview of ink private void pnlInk_Paint(object sender, PaintEventArgs e) { if (strksDisplay != null) { Renderer r = new Renderer(); // Offset the ink to be drawn so it will display in // the top-left of the preview window Rectangle rcBounds = strksDisplay.GetBoundingBox(); r.Move(-rcBounds.Left, -rcBounds.Top); // Figure out how much the ink needs to be scaled in // order for it all to fit within the preview window float fScaleWidth = 1, fScaleHeight = 1; Misc.InkSpaceRectToPixels(r, e.Graphics, ref rcBounds); if (pnlInk.ClientSize.Width < rcBounds.Width) { fScaleWidth = (float)pnlInk.ClientSize.Width / rcBounds.Width; } if (pnlInk.ClientSize.Height < rcBounds.Height) { fScaleHeight = (float)pnlInk.ClientSize.Height / rcBounds.Height; } float fScaleBy = Math.Min(fScaleWidth, fScaleHeight); r.Scale(fScaleBy, fScaleBy); // Now we can draw the ink r.Draw(e.Graphics, strksDisplay); } } // Handle copying the current result to the clipboard private void btnCopy_Click(object sender, EventArgs e) { Clipboard.SetDataObject(txtResults.Text); } } } The dialog consists of a read-only TextBox containing the recognized text for the current top alternate, a ListBox containing alternates for the current selection, and a Panel where the strokes constituting the current selection is drawn. As the selection is changed in the TextBox, the alternates and stroke preview are updated. If an alternate is selected, the text in the TextBox is updated accordingly.
The ListBox of alternates is filled with RecognitionAlternate instances obtained from the GetAlternatesFromSelection method:
// Fill up the listbox with each alternate foreach (RecognitionAlternate recoAlternate in recoAlternates) { lbAlternates.Items.Add(recoAlternate); } When an alternate is selected, the real functionality of the sample is exhibited. The ModifyTopAlternate method of the RecognitionResult class allows the top alternate to be overridden, in essence correcting it. No matter what the granularity of the alternate (be it multiple words or a single segment), ModifyTopAlternate adjusts the recognition result accordingly:
// Get the alternate that was selected RecognitionAlternate recoAlternate = (RecognitionAlternate)lbAlternates.SelectedItem; // Adjust the current reco results with it recoResult.ModifyTopAlternate(recoAlternate);
When the Copy button is clicked, the recognition result text is placed onto the clipboard and the dialog is closed. If the user chooses the Copy To Clipboard menu item again, the dialog will show the modified RecognitionResult instance, ready for further use or correction.
If ink is added to or removed from the document, a new RecognitionResult instance is generated by the recognizer this means that any previous user modifications to the top alternative are lost. This is an understandable but unfortunate reality, but it is possible to overcome it. One method is to remember the top alternative string and match it against the new reco result; unfortunately, this is a complex algorithm, and it would likely be too performance intensive. Another way is to use a capability of the Strokes collection it can be tagged with recognition results. If you are able to divide your document into separate Strokes collections (perhaps by every line or paragraph), only the results directly affected by adding or removing ink will be reset. The question then arises, How do we associate recognition result instances to Strokes collections? The next section explains how.
Storing Recognition Results
Once a recognition result instance has been obtained and optionally modified by the user, it can be useful to persist those results along with the Ink object that contains the recognized strokes. Alternatively, it can be useful to associate a Strokes collection to a recognition result instance for a variety of reasons, including splitting an ink document into pieces to preserve edited recognition results and maintaining recognition results for multiple languages simultaneously.
The Strokes collection class possesses a RecognitionResult property, used to reference a RecognitionResult instance. It is set from the SetStrokesResult method found on the RecognitionResult class. Using explicit assignment to set the RecogitionResult property on a Strokes collection will cause an exception to be thrown. Only the SetStrokesResult method is able to alter the property s value, in order to ensure consistency between the strokes referenced by the collection and the RecognitionResult.
After setting the RecognitionResult property of a Strokes collection, an application will typically hang onto that Strokes collection for future use perhaps as long as the recognition results are needed. You might think this is not necessary because subsequent Strokes collections that reference the same ink strokes as a Strokes collection with a RecognitionResult instance will reference the RecognitionResult. This is not so! The reference is kept only on a per-instance basis of a Strokes collection. Consider the following code:
// Recognize all ink on the page RecognizerContext recoCtxt = recognizer.CreateRecognizerContext(); recoCtxt.Strokes = inkCtl.InkOverlay.Ink.Strokes; recoCtxt.EndInkInput(); RecognitionResult recoResult = recoCtxt.Recognize(); // Set the results on the recoCtxt's RecognitionResult property recoResult.SetStrokesResult(); RecognitionResult recoResult2 = inkCtl.InkOverlay.Ink.Strokes.RecognitionResult;
The recoCtxt.Strokes.RecognitionResult will reference the same RecognitionResult instance as recoResult because of SetStrokesResult being called. However, the recoResult2 reference will be null and definitely not equivalent to recoResult. Recall from Chapter 5 that the Strokes property of the Ink class always returns a copy of the collection, so the RecognitionResult property would not be set.
The contents of a RecognitionResults instance can be persisted by adding a Strokes collection referencing the recognition results to the CustomStrokes collection on the Ink class. Storing recognition results is useful for at least two reasons: recognition results don t have to be recomputed when a file loads avoiding the application freezing during synchronous reco or not behaving optimally during asynchronous reco and down-level support machines that don t have a recognizer installed can still view the alternates that were computed.
Recognition Properties
You ve seen how quite a lot of data can be extracted from a recognition result by using the computed alternates. Even more data can be obtained by using APIs supporting recognition properties. Recognition properties are Guid values that represent various data found in a recognition alternate and provide a generic means of accessing and specifying data. The RecognitionProperty class contains the values, listed in Table 7-9, of the supported recognition properties.
Value | Description |
ConfidenceLevel | The confidence level the recognizer has in the recognition result |
HotPoint | The hot point of the recognition; typically used by gesture recognizers |
LineMetrics | The line metrics of the recognition result |
LineNumber | The line number of the result; not supported for recognizers of East Asian characters |
MaximumStrokeCount | The maximum stroke count for the recognition result |
PointsPerInch | The points-per-inch metric for the recognition result |
Segmentation | The ink fragment or unit the recognizer used to produce the recognition result |
The RecognitionAlternate class s AlternatesWithConstantPropertyValues method chops an alternate into one or more alternates that are grouped by the same specific property value. It accepts a RecognitionProperty Guid value and returns a RecognitionAlternates collection containing the appropriate RecognitionAlternate instances.
Using the AlternatesWithConstantPropertyValues method can yield some cool information. For example, it becomes easy to underline inked words in colors according to the recognized confidence level by chopping an alternate into the segments as computed by the recognizer. A green underline can indicate strong confidence, yellow intermediate, and red poor. The code would look something like:
// Underline words in a color according to their confidence // Green strong, Yellow intermediate, Red - poor Graphics g = inkCtl.InkInputPanel.CreateGraphics(); foreach (RecognitionAlternate alt in recoResult.TopAlternate.AlternatesWithConstantPropertyValues( RecognitionProperty.Segmentation)) { // Skip over any whitespace if (alt.Strokes.Count == 0) { continue; } // Get start point of the segment's baseline Point ptStart = alt.Baseline.BeginPoint; RendererEx.InkSpaceToPixel(r, e.Graphics, ref ptStart); // Get end point of the segment's baseline Point ptEnd = alt.Baseline.EndPoint; inkCtl.InkOverlay.Renderer.InkSpaceToPixel(g, ref ptEnd); // Draw the baseline in a color according to confidence Pen pen = null; switch (alt.Confidence) { case RecognitionConfidence.Strong: pen = Pens.Green; break; case RecognitionConfidence.Intermediate: pen = Pens.Gold; break; case RecognitionConfidence.Poor: pen = Pens.Red; break; } g.DrawLine(pen, ptStart, ptEnd); } g.Dispose(); You can see this functionality in action by modifying any of the samples that maintain a RecognitionResult instance, and then overriding the InkInputPanel s Paint event handler. The information can be a handy visual indicator of results that don t have a high confidence, letting users know they should double-check the result from the correction UI.
The results of calling AlternatesWithConstantPropertyValues with RecognitionProperty.ConfidenceLevel and RecognitionProperty.LineNumber are exposed through the ConfidenceAlternates and LineAlternates properties, respectively. They are provided as properties solely for convenience.
Some of the recognition properties don t make much sense for use with the AlternatesWithConstantPropertyValues method for example, the RecognitionProperty.MaximumStrokeCount and RecognitionProperty.PointsPerInch properties. They are meant to be used with the other method that supports recognition properties, namely the GetPropertyValue method. This is used to serialize recognition data to a byte array.
Improving Recognition Results
Recognition accuracy can be improved by giving the recognizer information about the expected input. The Tablet PC Platform supports this information in two forms: hints about the content and hints about the layout or positioning of the ink.
Factoids Content Hint
As mentioned earlier in the chapter, factoids are content hints given to the recognizer so that it can bias its results and achieve greater accuracy. Sometimes the context or type of information being entered is known by an application for example, asking the user to enter a phone number. By specifying to the recognizer that the ink it will perform recognition on probably represents a phone number, the application can assume a certain format and distinguish between letters and numbers (such as 0 or O, and 1 or I).
A Factoid About Factoids
Norman Mailer first coined the term factoid in his biography of Marilyn Monroe. In the biography, Mailer intended factoid to mean a false piece of information that everyone assumes is true because it appears in print. In Marilyn Monroe s case, the press would often invent factoids about her to create drama and interest.Factoid is a misnomer as used by the Tablet PC Platform SDK. When used during ink recognition, a factoid is really a factette: a small piece of interesting trivia.
The Tablet PC Platform supports numerous kinds of factoids, which are listed in Table 7-10. Different languages can have different conventions for the content format for example, phone numbers in North America and in Europe, and country postal codes. Some recognizers don t support some factoids, as noted in the table.
The Factoid property of the RecognizerContext class is used to specify the factoids the recognizer should use. Factoids are specified by a string, the values as seen in the left-hand column of Table 7-10. Multiple factoids are specified by separating them with a pipe character ( ). Some examples of setting factoids are thus:
// Recognize a date recoCtxt.Factoid = "DATE"; // Recognize an email address or URL recoCtxt.Factoid = "EMAIL WEB"; // Restore the factoid back to default behavior recoCtxt.Factoid = "DEFAULT";
Factoid Value | Description | Example/Notes |
DIGIT | Bias toward single digits | 0, 1, 2, 3, etc. |
Bias toward e-mail addresses | jimmy@adatum.com | |
WEB | Bias toward URLs | http://www.microsoft.com www.microsoft.com/tabletpc |
DEFAULT | Returns the recognizer to default settings | n/a |
NONE | Disables all factoids, dictionary, and language model | n/a |
FILENAME | Bias toward file name path | C:\ Chapter7.doc |
SYSDICT | Enables the system dictionary | n/a |
WORDLIST | Enables the word list (see the following section) | n/a |
CURRENCY | Bias toward currency: dollars, euros, yen, pounds, etc. | $100.00 |
DATE | Bias toward dates | 11/7/03 |
NUMBER | Bias toward numbers, including math symbols, ordinals, suffixes such as KB and MB, and the TIME and CURRENCY factoids | 456 |
ONECHAR | Bias toward a single ANSI character | A, B, C, D, 0, 1, 2, |
PERCENT | Bias toward a number followed by the percent symbol | 100 % |
POSTALCODE | Bias toward postal codes | 98052 |
TELEPHONE | Bias toward telephone numbers | 555-2368 (425)555-2368 |
TIME | Bias toward time | 3:38 pm |
UPPERCHAR | Bias toward a single uppercase character | A, B, C, D, etc. |
KANJI_COMMON | Bias toward Kanji characters; depends on language for exact set | East Asian only |
JPN_COMMON | Bias toward KANJI_COMMON, HIRAGANA, KATAKANA, alpha numeric, standard punctuation, and symbol characters | Japanese only |
CHS_COMMON | Bias toward KANJI_COMMON, alphanumeric, standard punctuation, and symbol characters | Chinese (simplified) only |
CHT_COMMON | Bias toward KANJI_COMMON, BOPOMOFO, alphanumeric, standard punctuation, and symbol characters | Chinese (traditional) only |
KOR_COMMON | Bias toward KANJI_COMMON, JAMO, HANGUL_COMMON, alphanumeric, standard punctuation, and symbol characters | Korean only |
HIRAGANA | Bias toward Hiragana characters | Japanese only |
KATAKANA | Bias toward Katakana characters | Japanese only |
BOPOMOFO | Bias toward Taiwanese phonetic characters | Chinese (traditional) only |
JAMO | Bias toward Jamo characters | Korean only |
HANGUL_COMMON | Bias toward Hangul characters | Korean only |
You can consult the Tablet PC Platform SDK documentation for more details on what exact formats and characters are allowed for various languages.
Word Lists Content Hint
Factoids do a great job for common types of data, but what about application-specific types, for example, coworkers names or parking lot section numbers? Table 7-10 lists a factoid type named WORDLIST that instructs the recognizer to use a predefined set of words that are likely to appear in the recognized text. The RecognizerContext class s property WordList is a collection of strings the recognizer should bias toward. Like other factoids, word lists can either be suggested or forced; by default, word lists are suggested, but if the recognizer context flag RecognitionModes.Coerce is specified the results are forced to match the word list contents.
Determining if a Word Is Supported
After setting up a factoid, you might want to determine whether the recognizer context is in a state to recognize a specific word. The RecognizerContext class method IsStringSupported provides this information.
Guides Spatial Hint
The hints we ve been discussing so far have been content related. Recognizer guides specify the layout or positioning of ink. An electronic form might have boxes in which to enter characters of a name. Text entry UI such as the Tablet PC Input Panel has a well-defined area in which to draw ink. Recognizer guides allow an application to tell the recognizer where to expect ink characters or words to be drawn.
The RecognizerContext class s property Guide is used to specify an instance of RecognizerGuide to use. The constructor of the RecognizerGuide class is given the number of rows and columns in the guide, the midline height for ink (the distance between the baseline and the middle of a character), the writing area rectangle, and the drawing area rectangle. Figure 7-4 illustrates how these properties relate to one another.
Figure 7-4. RecognizerGuides provide the ability to indicate the layout of text that is entered. This example shows two rows by one column of text.
Character auto-complete functionality requires a guide to be set for it to function properly. This is because the recognizer needs to know the positioning and scale of strokes as they are entered in order to provide the best results.
EAN: 2147483647
Pages: 73