Technical Considerations
Technical Considerations
Now that we re done with the reasons you might want to revise your application to leverage the Tablet PC Platform, we re ready to evaluate the technical considerations involved with using the SDK. These considerations are grouped into the two broad categories of application design and performance.
Application Design
Successful integration of Tablet PC Platform features will require some planning in regard to an application s design. Several key design issues are surveyed in the following sections. You might find that while some of these issues might not concern your application, others might dramatically affect how you end up implementing your pen and digital ink features.
InkCollector and InkOverlay
Assimilating either an InkCollector object or an InkOverlay object into an application can be easy, depending on what you have in mind. For the purposes of this discussion, we will assume that you are using InkOverlay, though everything in this section applies to InkCollector as well. The simplest type of application that integrates InkOverlay is one in which there is both a mode and an area that accepts ink. For instance, if an application has an Insert Inking Surface feature that inserts a window that collects ink only when it s selected and activated, it is fairly straightforward to use InkOverlay with that window. Similarly, if an application has an annotation mode that collects ink annotations over a document, it is easy to enable and disable InkOverlay as needed.
However, one of the challenges of integrating InkOverlay comes in making the boundary between inking and other operations modeless. There are a few obstacles to overcome if you want there to be no explicit mode switching between the user inking and doing other things (for instance, selecting or bringing up a context menu).
One such obstacle is that tablet input events and mouse messages come in an unpredictable order. The reasons for this were described in detail in Chapter 4. The effect, nonetheless, is that your application s existing input message handler might need to be made aware of tablet input events (and their unpredictable order) to respond correctly to user action. There might be a variety of cases in which the mixture of input messages and tablet input events yield unexpected behavior in your application, and some of them might be tricky to debug.
Another potential hurdle when implementing modeless inking is that you will most likely want to respond to many of the events sent by InkOverlay in order to override or cancel them. For instance, when the user touches the pen to the window, you ll want to first decide whether the pen might be targeting a UI element such as a selection handle. If so, you ll likely cancel the stroke in progress (by making it transparent and later discarding it when the Stroke event fires, for example) and instead track the dragging of the handle. There are many other cases in which you might need to listen to tablet input events in order to override or augment the default behavior.
In the end, you might find that it s easier to control both when and where the user can ink by supplying an explicit inking mode and an inking window or area. A modeless interface is more difficult to get right, and it can even be frustrating if it s not tweaked to respond well to the average user s expectations.
Storage
Your application probably already stores its data in a particular way. Depending on how closely compatible it is with your existing format, it shouldn t be too difficult for you to integrate the Tablet PC Platform s digital ink storage format into your application. The Tablet PC stores digital ink in four flavors: Ink Serialized Format (ISF), Graphics Interchange Format (GIF), and Base64-encoded variants of both.
ISF is the default format for storing digital ink on the Tablet PC Platform. It is an opaque (not publicly documented) binary format that stores all the relevant data from an Ink object so that the same ink can be reconstituted later. It is also the standard format by which digital ink is exchanged between applications from the Clipboard on Tablet PCs. ISF supports two modes of compression in order to reduce the size of the resulting binary stream.
GIF is the once-popular 256-color image format supported by all Internet browsers and most image applications. You might be struck by the seeming oddity of the Tablet PC Platform supporting GIF, of all things, for storage. GIF was chosen as the down-level compatibility solution for digital ink (that is, the solution for the problem of how non-Tablet PC users can view digital ink created on a Tablet PC). Two characteristics make GIF a good solution: its widespread support and its ability to embed arbitrary metadata. ISF is actually embedded in GIF s metadata section, thereby making it possible to recapture the original ink by using the SDK to load such a GIF.
Base64 is a method of encoding binary streams into text. The Tablet PC Platform supports saving of its ISF and GIF formats in Base64-encoded flavors, thereby making it possible to embed the streams into text-based formats such as XML and HTML. We will analyze the amount of bloat that Base64 encoding introduces in the upcoming section on performance.
In all likelihood your application data is already being stored in some convenient binary or text format. It should be straightforward to store digital ink alongside your current data using the Tablet PC Platform s Ink.Save method.
One final thing to think about is versioning of the storage format, which the Tablet PC Platform documentation does not explicitly address. The problem is with both backward compatibility (future versions reading current formats) and forward compatibility (current versions reading future formats) because the platform guarantees neither. Although it might seem reasonable to expect that at least backward compatibility will be preserved, lack of an explicitly stated policy or guarantee should make you wary. Do not assume that future revisions of the platform will handle versioning transparently! If your application needs to guarantee backward or forward compatibility, you might need to provide that functionality yourself by storing version information, for example.
Clipboard
Your application probably interacts with the Clipboard to interoperate with other applications. There are three ways in which you can incorporate digital ink into your existing Clipboard code:
- Embed it into your custom Clipboard format
If you put a custom data format on the Clipboard specific to your application, you can embed ISF (either binary or Base64) as part of that format. Your application will presumably be responsible for both sides of the interchange (putting the custom format on the Clipboard and reading it off).
- Embed it into a well-known Clipboard format
HTML, RTF, WMF, BMP, and Unicode are just some of the many well-known Clipboard formats that popular applications expect. When embedding digital ink in HTML, use the GIF format supported by the platform to maximize interoperability. WMF and BMP can be generated simply by drawing the ink using the appropriate Graphics object, while Unicode can be obtained from a RecognitionResult s TopAlternate property. Whether your application reads these well-known formats is up to you.
- Put it directly on the Clipboard
Ink objects themselves will interact with the Clipboard if you want them to. Using methods such as ClipboardCopy, CanPaste, and ClipboardPaste, you can easily put digital ink on the Clipboard and read it off as well. This makes for an easy way to get up and running with the Clipboard from the Tablet PC Platform.
NOTE
One caveat concerning putting ink directly on the Clipboard: the IDataObject created by ClipboardCopy cannot have other data formats added to it! If you want the Clipboard to have native ink data as well as some of your application s own data on it, you ll have to create an IDataObject of your own.
Undo
The ability to undo actions has become standard across applications, and most users assume its existence as a matter of course. Undo is one of those features that seems harmless and simple at the outset, but often turns out to be tremendously complicated to implement correctly. Unfortunately, the Tablet PC Platform SDK offers almost no help whatsoever to make the task of implementing undo any easier. Let s take a moment to first understand the problem.
Applications that support undo functionality typically support either single-level undo or multilevel undo. Single-level undo allows the undoing of only one action the one most recently performed and might support redoing of that same action once undone. Multilevel undo, on the other hand, allows undoing of more than one action in sequence, usually back to the last point you saved or to when you first launched the application.
In addition to supporting either single-level or multilevel undo, advanced applications might also support undoing of actions across control or process boundaries. For instance, if you type a sentence in Microsoft Word, embed a Microsoft Excel spreadsheet into the document, and edit the spreadsheet as well, you ll find that you can actually sequentially undo actions performed across both Word and the embedded Excel in the order you performed them. This is natural and expected, until you actually try to figure out what s going on! How is it that Word tells Excel when to undo an action? How does Excel s undo information live in Word, since they re different applications running in different processes? The answer, perhaps surprisingly, is not that it s a Microsoft thing. In fact, there is nothing proprietary going on. Both Word and Excel (among many other applications and Microsoft ActiveX controls) support the Object Linking and Embedding (OLE) Undo standard, which defines a protocol for applications to share undo information with each other. Whenever a user action is performed, the application is responsible for generating the undo information, called an undo unit, required to fully undo that action. These undo units are then stored by the host application and executed during an undo operation. Any application supporting OLE Undo can thus embed and undo the actions of any other OLE Undo-compatible control or application that it might interact with, thereby creating a seamless undo experience for the user. Figure 9-1 gives an example of what such an undo stack might look like.
Figure 9-1. One possible way to visualize the undo units in Microsoft Word s undo stack.
Other than the InkPicture control, which has an Undo method supporting single-level undo, the Tablet PC Platform SDK is entirely without undo functionality. In fact, in some ways the SDK even makes implementing your own undo support difficult.
The heart of the problem is that the SDK does not generate undo units. Well, you astutely remark, I ll just generate my own! If only it were so easy. You could generate undo units yourself if you knew what information to put in them. For some of the methods in the SDK, the required information is readily available. For example, if you create a stroke from CreateStroke, it s easy enough to store away the resulting Stroke so that during undo you can call DeleteStroke on it. However, other methods in the SDK make it difficult, and sometimes impossible, to deduce the necessary information to cleanly undo an action. Take, for instance, the Transform method. You can apply an arbitrary transform to a set of strokes, but what if you wanted to restore its original transform during an undo? What was the original transform? The SDK does not give you a way to determine it once it s been changed from identity (the default transform)! In this case, you d have to separately remember the transforms you ve applied to an ink object in case you ever have to undo them. It might not at first strike you as tedious, but in trying to work around these deficiencies you will soon discover that there is a lot that needs to be done. When calling Ink.ExtractStrokesAtRectangle, how do you determine which original strokes were affected so that you can later reassemble them? How would you undo a Clip operation? Many of these issues are surmountable, but most only with reasonable difficulty.
Ah! you conclude, I ll just clone Ink objects whenever I change them so that I can reinstitute their clones when undoing an action. Indeed, this is a viable solution, particularly if your application only needs single-level undo. It s also fairly simple to implement, once you ve isolated all the code that modifies Ink objects directly (or indirectly through Strokes or other means). With good design this is doable. The solution can also be extended to support multilevel undo by creating as many clones as necessary whenever Ink objects are changed. Before latching onto this solution, however, the outstanding issues of performance and resources need to be resolved. Cloning Ink objects during modification can be both slow and memory intensive if not done with restraint. Particularly when Ink objects contain a lot of strokes, the amount of time and memory required can be prohibitive. The situation is further exacerbated if you require multilevel undo.
There is no easy solution to the problem of undo. We recommend that you fully investigate and understand the implications of any solution you consider because many snares lie in wait for the unsuspecting! Later in the chapter we present a sample application that implements a simple undo architecture by cloning Ink objects.
Ink Architecture
When designing your application, the question of its underlying ink architecture will undoubtedly arise. By ink architecture, we mean the way in which you will represent ink internally in your application s data structures. One of the decisions to be made is whether you will have one Ink object containing all the digital ink in the document or whether you will instead have a set of smaller Ink objects each containing a portion of the total ink. Recall from Chapter 5 that this was referred to as Big Ink vs. Little Ink. In some cases, the choice between these two architectures will fall naturally out of the ink features themselves. For instance, if you want the user to be able to attach ink annotations to various objects in your document, it might make a lot of sense to have each annotation consist of its own Ink object. Or you might have one big Ink object if instead your application has a dedicated scribbling area for the user.
The choice between using one big Ink object versus many little Ink objects might be inconsequential given your needs, in which case you should pick the architecture that s most convenient. It s usually easy to transfer strokes between Ink objects or to operate only on select strokes, further diminishing the distinction between the architectures. Sometimes, however, one architecture will have clear advantages over the other. If you support multilevel undo by cloning Ink objects, having a set of little Ink objects might be the right design to reduce memory usage because you would then have to clone only the Ink objects that are actually changed by a particular operation. Alternatively, if you have a simple inking surface that doesn t support undo, using one big Ink object might be easiest. Other contributing factors specific to your application are likely to influence your architecture as well.
Accessibility
One final issue to consider is that of accessibility. In particular, ink should respect the high contrast settings in Windows. Fortunately, the InkPicture, InkCollector, and InkOverlay classes all support high contrast mode through their SupportHighContrastInk property (which defaults to true). When using these objects, nothing additional needs to be done to comply with accessibility requirements ink strokes will automatically paint in high contrast (usually white on a black background, or whatever the system-defined high contrast colors are). However, the Renderer class doesn t have built-in support for high contrast. Unfortunately, this means that if you ever draw ink using a Renderer object (which is all too likely if you use Ink objects at all), you ll have to manually specify DrawingAttributes appropriately prior to rendering to achieve high contrast and accessibility compliance. This will require that you determine the system s high contrast colors if high contrast is turned on, and set the appropriate color in the DrawingAttributes passed to Renderer.Draw.
Performance
No discussion on integrating Tablet PC Platform features would be complete without considering performance. Digital ink is captured by the platform at high resolution and rendered with a fair degree of realism. These and other features of the platform come with a performance cost that you should be aware of. At the same time, with a good understanding of the platform s various performance trade-offs you will be able to strike the right balance between the features you need and the performance you want.
We divide our analysis of performance into two dimensions: time and space. First we consider how fast (or slow) various key operations on the platform are. Next we take a look at how much memory and disk space various platform features require. In the following sections we provide a simple framework in which performance in a few key areas can be tested. You will no doubt want to independently analyze performance characteristics specific to your application.
The performance analysis given here is minimal and in many cases imprecise. One point to keep in mind is that the code samples used to collect this performance data are designed foremost for clarity. There are many complex issues (such as cache latency, paging/swapping, and video memory) that have been intentionally ignored in our analysis. The performance numbers are best used as ballpark figures to aid in decision making. They are perhaps most reliable as measurements of relative performance between features of the platform, not as absolute gauges of performance you should expect on all Tablet PCs.
Speed of Common Operations
As an introduction to the performance characteristics of the managed API, we include the LoadSaveRenderPerf application that tests the speed of loading, saving, and rendering of digital ink. Some simplifying assumptions have been made in the application in order to keep the code as lucid as possible. We will discuss these assumptions after presenting the code. Figure 9-2 shows the application just after profiling rendering.
Figure 9-2. The LoadSaveRenderPerf application tests the performance of loading, saving, and rendering using the managed API.
LoadSaveRenderPerf.cs
//////////////////////////////////////////////////////////////////// // LoadSaveRenderPerf.cs // // (c) 2002 Microsoft Press, by Philip Su // // This program measures load, save, and render performance //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using System.IO; using Microsoft.Ink; public class LoadSaveRenderPerf : Form { private Panel m_pnlRender; private Button m_btnLoad; private Button m_btnSave; private Button m_btnRender; private Button m_btnRenderBasic; private Label m_label; private Ink m_ink; // Cached ink, once loaded private int m_cStrokes; // Number of strokes in m_ink private int m_cAvgPoints; // Avg num points per stroke struct RenderParams // Parameters for the Render { // perf runs. public Renderer renderer; public Graphics graphics; public Ink ink; }; // Name of the ISF file we will load our data from. Be // sure that this file is in the directory that this // application executes from! const string s_strFileName = "PerfInk.isf"; // Entry point of the program static void Main() { Application.Run(new LoadSaveRenderPerf()); } // Main form setup public LoadSaveRenderPerf() { bool fLoadSuccessful = false; CreateUI(); // Create our dialog // Now that the dialog is up and running, attempt // to load our ISF file. try { FileStream fs; int cbRead = 0; // Count of bytes read byte[] rgb; // Our ISF bytes from disk // Attempt to load the ISF file. Make sure that // this file is in the directory you run this // application from! fs = new FileStream(s_strFileName, FileMode.Open); rgb = new byte[fs.Length]; cbRead = fs.Read(rgb, 0, rgb.Length); // Read ISF if (cbRead == fs.Length) { m_ink = new Ink(); m_ink.Load(rgb); // Load the ISF into ink CalcInkStatistics(); // Figure out our stats fLoadSuccessful = true; } fs.Close(); } catch (FileNotFoundException) { } if (!fLoadSuccessful) { MessageBox.Show("File " + s_strFileName + " not loaded successfully. Please " + "make sure it is in the same " + "directory as this program.", "Load error"); m_ink = null; } } // Instead of throwing in the constructor when loading // fails, we simply wait for the dialog to be shown, at // which point we close if loading failed. protected override void OnVisibleChanged(EventArgs e) { if (m_ink == null) // Must not have loaded it! Close(); } // Load button handler. private void OnLoad(object sender, EventArgs e) { const int kcReps = 50; // Number of times to repeat double sec = 0.0; // Total time byte[] rgb; // The bytes we should load from // Generate our bytes to load from m_ink rgb = m_ink.Save(); // Run the perf test sec = TestPerf(new RunSingle(LoadRunSingle), kcReps, rgb); ShowResults("loads", kcReps, sec); } // Save button handler. private void OnSave(object sender, EventArgs e) { const int kcReps = 50; // Number of times to repeat double sec = 0.0; // Total time sec = TestPerf(new RunSingle(SaveRunSingle), kcReps, null); ShowResults("saves", kcReps, sec); } // Render button handler. private void OnRender(object sender, EventArgs e) { // Render smooth ink (passing true for fSmooth) TestRenderPerf(GetRenderingInk(true)); } // Render Basic button handler. private void OnRenderBasic(object sender, EventArgs e) { // Render simple ink (passing false for fSmooth) TestRenderPerf(GetRenderingInk(false)); } // Display the results from a particular run of cReps // repetitions lasting sec seconds. private void ShowResults(string strWhat, int cReps, double sec) { m_label.Text = string.Format("{0} {1} of {2} strokes " + "averaging {3} points each took {4:0.00} " + "seconds \n({5:0.} strokes/sec)", cReps, strWhat, m_cStrokes, m_cAvgPoints, sec, (m_cStrokes * cReps) / sec); } // Stores the number of ink strokes into m_cStrokes, and // calculates the average number of points per stroke. private void CalcInkStatistics() { int cTotalPoints = 0; m_cStrokes = m_ink.Strokes.Count; if (m_cStrokes > 0) { foreach (Stroke s in m_ink.Strokes) cTotalPoints += s.PacketCount; m_cAvgPoints = cTotalPoints / m_cStrokes; } else { m_cAvgPoints = 0; } } // Tests rendering perf. Sets up the RenderParams // with a Renderer and a Graphics object, as well as // the ink object that should be rendered. private void TestRenderPerf(Ink ink) { const int kcReps = 10; // Number of times to repeat double sec = 0.0; // Total time RenderParams rp; rp = new RenderParams(); rp.renderer = new Renderer(); rp.graphics = m_pnlRender.CreateGraphics(); rp.ink = ink; sec = TestPerf(new RunSingle(RenderRunSingle), kcReps, rp); rp.graphics.Dispose(); // Clean up our graphics object ShowResults("renders", kcReps, sec); } // Clones m_ink and changes all drawing attributes to be // either smooth or not. "Smooth" is defined as being both // antialiased and Bezier curve fitted. Also scales the // ink to fit within m_pnlRender; private Ink GetRenderingInk(bool fSmooth) { Ink ink = m_ink.Clone(); // Make a copy of the ink // Modify each stroke's drawing attributes accordingly foreach (Stroke s in ink.Strokes) { DrawingAttributes drawattrs = s.DrawingAttributes; drawattrs.AntiAliased = fSmooth; drawattrs.FitToCurve = fSmooth; s.DrawingAttributes = drawattrs; } // Scale the ink to fit within m_pnlRender if (ink.Strokes.Count > 0) { Renderer renderer = new Renderer(); Graphics graphics = m_pnlRender.CreateGraphics(); Point ptExtent = new Point(m_pnlRender.Bounds.Size); Rectangle rcInk; // The new rectangle for ink to fit renderer.PixelToInkSpace(graphics, ref ptExtent); rcInk = new Rectangle(0, 0, ptExtent.X, ptExtent.Y); ink.Strokes.ScaleToRectangle(rcInk); graphics.Dispose(); // Free our graphics resources } return ink; } // Our delegate allows all four perf tests to use the // same timing function TestPerf(). Each RunSingle // delegate is responsible for doing a single iteration // of the perf test (whether it be load, save, or render) private delegate void RunSingle(object o); private void LoadRunSingle(object o) { byte[] rgb = (byte[])o; // We're passed a byte array Ink ink = new Ink(); // See text for important note! ink.Load(rgb); } private void SaveRunSingle(object o) { m_ink.Save(); } private void RenderRunSingle(object o) { RenderParams rp = (RenderParams)o; // Clear ourselves first so that progress is visible. // This impacts the resulting timing negligibly. m_pnlRender.Invalidate(); m_pnlRender.Update(); rp.renderer.Draw(rp.graphics, rp.ink.Strokes); } // The heart of our performance testing. Runs cReps // repetitions of "single", the delegate that actually // performs the profiled functionality. private double TestPerf(RunSingle single, int cReps, object oParam) { DateTime dtStart; // Set our "running" text. Need to Update() right // away, otherwise the form is too busy to repaint. m_label.Text = "Testing performance -- please wait..."; m_label.Invalidate(); m_label.Update(); dtStart = DateTime.UtcNow; // Record starting time // Run the tests on our delegate's method for (int i = 0; i < cReps; i++) single(oParam); // Return the time delta since we began return (DateTime.UtcNow - dtStart).TotalSeconds; } const int cSpacing = 8; // Spacing between controls private void CreateUI() { int x = cSpacing; // Position of next control int y = cSpacing; SuspendLayout(); // Create and place all of our controls. You may // wish to experiment with setting m_pnlRender.Size // to something bigger, like 900x650, since it // greatly affects render performance. m_pnlRender = new Panel(); m_pnlRender.BorderStyle = BorderStyle.FixedSingle; m_pnlRender.Location = new Point(x, y); m_pnlRender.Size = new Size(400, 192); // m_pnlRender.Size = new Size(900, 650); y += m_pnlRender.Bounds.Height + cSpacing; CreateButton(ref m_btnLoad, ref x, y, 50, 24, "&Load", new EventHandler(OnLoad)); CreateButton(ref m_btnSave, ref x, y, 50, 24, "&Save", new EventHandler(OnSave)); CreateButton(ref m_btnRender, ref x, y, 70, 24, "&Render", new EventHandler(OnRender)); CreateButton(ref m_btnRenderBasic, ref x, y, 100, 24, "Render &Basic", new EventHandler(OnRenderBasic)); x = cSpacing; y += m_btnRenderBasic.Height + cSpacing; m_label = new Label(); m_label.BorderStyle = BorderStyle.FixedSingle; m_label.Location = new Point(x, y); m_label.Size = new Size(m_pnlRender.Width, 32); m_label.Text = "Press any button to begin"; // Configure the form itself AutoScaleBaseSize = new Size(5, 13); ClientSize = new Size(m_label.Right + cSpacing, m_label.Bottom + cSpacing); Controls.AddRange(new Control[] { m_pnlRender, m_btnLoad, m_btnSave, m_btnRender, m_btnRenderBasic, m_label }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "Load, Save, Render Performance"; ResumeLayout(false); } // Creates a button, updating the x location for the next // button. private void CreateButton(ref Button btn, ref int x, int y, int cWidth, int cHeight, string strName, EventHandler eh) { btn = new Button(); btn.Location = new Point(x, y); btn.Size = new Size(cWidth, cHeight); btn.Text = strName; btn.Click += eh; x += btn.Width + cSpacing; } } A large chunk of this code sets up the user interface, but we will focus on the non-UI elements of the application in our discussion. First, the application tests the performance of Ink.Load, Ink.Save, and InkRenderer.Draw using delegate methods that each performs one iteration of the functionality being probed (LoadRunSingle, SaveRunSingle, and RenderRunSingle). The application gets the ink from a specific file, PerfInk.isf, which is available in the companion content. This file is provided merely for your convenience in fact, you can create your own ink and use it for these tests if you want. The one that comes with the companion content is basic, consisting of about 1300 black ink strokes of various lengths stored in ISF format. Make sure when running the application that PerfInk.isf is in the same directory as the executable file so that the ink can be successfully loaded. When the application starts, it loads the ink from PerfInk.isf and uses it for all the tests.
When the user taps a button, the application runs a number of iterations of the particular function being tested. You should feel free to modify the number of iterations to your liking. The default iterations were chosen to last between 10 20 seconds on an average 1-GHz Pentium 4 to give the trials the amount of time needed to get consistent results.
Lastly, the application tests both regular rendering and basic rendering, neither of which uses pressure data. The difference between these two modes is that basic rendering turns off antialiasing and curve fitting. With these two appearance enhancing features turned off, performance of rendering varies dramatically. Although you will likely keep antialiasing and curve fitting on when rendering ink, you might want to investigate the performance consequences of these features.
Enough about the application time to analyze the results! The following numbers were obtained on a 1.4-GHz Pentium 4 (Dell Dimension 8100) with 128-MB RAM running Windows XP Professional. The performance of the Tablet PC Platform SDK was as follows:
- Load: 18000 strokes per second
This number is somewhat affected by the code, which creates a new Ink object every time that Ink.Load is called. Unfortunately, there is no way to call Load multiple times on an Ink object because the SDK allows loading ink only into pristine (never dirtied) Ink objects. This result was thus obtained in tandem with creating 50 Ink objects (one per iteration). In addition, the ISF being loaded was saved using the platform defaults, which is normal compression. Surprisingly, loading both maximally compressed and uncompressed ink is neither faster nor slower than normal compression.
- Save: 8500 strokes per second
Ink.Save is much slower than Ink.Load using normal compression. Maximum compression has no effect on Save speed. On the other hand, if you use no compression, Save is twice as fast (19,000 strokes per second).
- Render: 600 strokes per second
Thin black ink was rendered with both antialiasing and curve fitting turned on, using an m_pnlRender size of 900 x 650 (instead of the code listing s 400 x 192). The larger size was used to simulate full-screen rendering because the size of the rendering surface is intimately tied to resulting performance. This result should be used only as a relative gauge of how long rendering takes. In addition to rendering surface size, the speed with which ink is rendered also depends greatly on factors such as the DrawingAttributes.PenTip used (Ball or Rectangle), whether pressure is available, and whether any transparency is used. You might want to investigate rendering speed using the type of digital ink expected by your application (incorporating factors such as width, average length, transparency, and pen tip) because there is such wide variance based on ink characteristics. Incidentally, the type of hardware on which you test rendering performance is also critical to the results, so choose carefully! Rendering hardware will greatly affect performance on Tablet PCs, especially if you re running in portrait mode. It s best to verify your application s rendering performance on a Tablet PC before finalizing your design.
- Basic render: 1200 strokes per second
Without antialiasing and curve fitting, the same ink renders twice as fast (while keeping the larger 900 x 650 m_pnlRender size). The dramatic difference in turning off the two ink-smoothing features might come as a surprise, but this result is typical on many hardware configurations. Antialiasing is often slower because it needs to read from video memory (an inherently slow operation on most hardware because video hardware is optimized for writing to, not reading from). Curve fitting fundamentally requires a greater number of calculations in order to find the B zier control points needed for rendering a smoother curve. Although you will typically retain these two ink-smoothing features for aestheticism, in the most dire performance situations you might want to revisit the decision.
By now you re probably thinking, What do these numbers mean? How much writing is, say, 1000 strokes worth? The answer depends on several variables, including the language being written as well as the handwriting style of the writer. In typical English, expect printed handwriting to be somewhere on the order of 6 strokes per word (about 170 words per 1000 strokes), while cursive writing is closer to around 1.6 strokes per word (or 625 words per 1000 strokes). Although there is much variance between these two ends of the English handwriting spectrum, note that the average cursive stroke is often around five times longer than the average printed stroke. This, in effect, brings both cursive and print to about the same number of packets per word. The performance estimates given above were obtained using printed handwriting strokes. Using those initial estimates, we d expect the same machine to render around 133 printed words per second. As a point of reference, a wide-rule 8.5-by-11-inch sheet of paper has around 30 lines. If each line has on average 10 words, rendering such a page will take a little over two seconds on the machine used in the preceding example. This ballpark estimate might be of great interest to you if you plan to develop an application that will support the user writing pages and pages of ink because it might feel quite slow flipping through such an application s document at two seconds a page.
In stark contrast to English, the 2965 most commonly used traditional Chinese characters have a weighted average of 9.1 strokes each. An average wide-rule line can easily accommodate 20 Chinese characters, further worsening the situation. All else being equal, we d expect a page of Chinese to take about seven seconds to render on the same machine used in the preceding example (disregarding its obviously shorter average stroke length). Toss in some transparency with pressure-variant ink and you ve got a real problem on your hands!
NOTE
Shih-Kun Huang (Institute of Information Science, Academia Sinica, Taiwan) and Chih-Hao Tsai (Council for Cultural Planning and Development, Taiwan) compiled the statistics in 1993 and 1994 regarding traditional Chinese characters. You can find other parts of the same study at http://www.geocities.com/hao510/charfreq/.
Fortunately, neither loading nor saving should be an issue because they are much faster and occur less often than rendering. Even several pages full of ink should, at these rates, take only a second or two to load or save. The performance trials were run by loading from and saving to memory so that speed of writing to the hard disk or other storage medium was not factored in. When deploying your application, the time required to access a hard disk might be significant and should be factored in.
Ink recognition is orders of magnitude slower than loading, saving, or rendering because it involves several sophisticated and computationally intensive algorithms. The English ink recognizer works at the pace of about 12 words per second. A full page of ink, using the familiar wide-rule 10 words per page assumption, will thus take on the order of 25 seconds to recognize a rather disconcerting conclusion. Figure 9-3 is a screenshot of the ink used to obtain these estimates (cursive writing yielded about the same performance). You can run the following sample application only on a machine running Windows XP Tablet PC Edition because handwriting recognition requires the presence of recognizers.
Figure 9-3. The RecoPerf application in action, recognizing several words from the Gettysburg Address.
RecoPerf.cs
//////////////////////////////////////////////////////////////////// // RecoPerf.cs // // (c) 2002 Microsoft Press, by Philip Su // // This program measures ink recognition performance //////////////////////////////////////////////////////////////////// using System; using System.Drawing; using System.Windows.Forms; using Microsoft.Ink; public class RecoPerf : Form { private Panel m_pnlInput; // The inking surface private ComboBox m_comboReco; // Which engine to use private Button m_btnReco; // Recognize button private Button m_btnClear; // Clear button private Label m_label; // Text private InkCollector m_inkCollector; // Ink collector object static void Main() { Application.Run(new RecoPerf()); } public RecoPerf() { CreateUI(); // Create a new InkCollector using m_pnlInput // for the collection area and turn on ink // collection. m_inkCollector = new InkCollector(m_pnlInput.Handle); m_inkCollector.Enabled = true; } // Handle the click of the Recognize button private void BtnRecoClick(object sender, EventArgs e) { const int cReps = 25; // Repeat the test 25 times Recognizer reco = (Recognizer)m_comboReco.SelectedItem; RecognizerContext context; RecognitionResult recoresult; RecognitionStatus recostatus; DateTime dtStart; int cSeconds = 0; int cWords = 0; string strResult; string strOutput; char[] rgchWhitespace = " \n\r\t".ToCharArray(); if (reco == null) { m_label.Text = "No recognizer selected!"; return; } // First recognize it once for our label's text context = reco.CreateRecognizerContext(); context.Strokes = m_inkCollector.Ink.Strokes; context.EndInkInput(); recoresult = context.Recognize(out recostatus); if (recostatus != RecognitionStatus.NoError) { m_label.Text = "There was an error while recognizing!"; return; } strResult = recoresult.ToString(); // First set text for user before testing reco speed m_label.Text = "Recognizing '" + strResult + "' " + cReps + " times..."; m_label.Invalidate(); m_label.Update(); // Run the actual recognition cReps times dtStart = DateTime.UtcNow; // Record starting time for (int i = 0; i < cReps; i++) { context.Recognize(out recostatus); } cSeconds = (int)(DateTime.UtcNow - dtStart).TotalSeconds; // A fairly simplistic calculation of the number of // words in the recognized string (western languages) cWords = strResult.Split(rgchWhitespace).Length; strOutput = string.Format("Recognizing {0} words {1} " + "times took {2} seconds", cWords, cReps, cSeconds); if (cSeconds > 0) { strOutput += " (" + (cWords * cReps) / cSeconds + " words/second)"; } m_label.Text = strOutput; } // Handle the click of the Clear button private void BtnClearClick(object sender, EventArgs e) { m_inkCollector.Ink.DeleteStrokes(); m_pnlInput.Invalidate(); } // Populate the recognizer combo box with all the // available recognizers private void LoadComboWithRecognizers() { Recognizers recos = new Recognizers(); Recognizer recoDefault = recos.GetDefaultRecognizer(); m_comboReco.BeginUpdate(); foreach (Recognizer r in recos) { m_comboReco.Items.Add(r); // Auto-select the default recognizer. // Unfortunately, Recognizer objects do not // implement Equals in a meaningful way, so we'll // live with comparing names. if (r.Name == recoDefault.Name) { m_comboReco.SelectedItem = r; } } m_comboReco.EndUpdate(); } private void CreateUI() { SuspendLayout(); // Create and place all of our controls m_pnlInput = new Panel(); m_pnlInput.BorderStyle = BorderStyle.Fixed3D; m_pnlInput.Location = new Point(8, 8); m_pnlInput.Size = new Size(600, 100); m_comboReco = new ComboBox(); m_comboReco.Location = new Point(8, 116); m_comboReco.Size = new Size(300, 24); m_comboReco.DropDownStyle = ComboBoxStyle.DropDownList; LoadComboWithRecognizers(); m_btnReco = new Button(); m_btnReco.Location = new Point(316, 116); m_btnReco.Size = new Size(80, 24); m_btnReco.Text = "&Recognize"; m_btnReco.Click += new EventHandler(BtnRecoClick); m_btnClear = new Button(); m_btnClear.Location = new Point(404, 116); m_btnClear.Size = new Size(50, 24); m_btnClear.Text = "&Clear"; m_btnClear.Click += new EventHandler(BtnClearClick); m_label = new Label(); m_label.Location = new Point(8, 148); m_label.Size = new Size(600, 24); m_label.BorderStyle = BorderStyle.FixedSingle; // Configure the form itself AutoScaleBaseSize = new Size(5, 13); ClientSize = new Size(616, 180); Controls.AddRange(new Control[] { m_pnlInput, m_comboReco, m_btnReco, m_btnClear, m_label }); FormBorderStyle = FormBorderStyle.FixedDialog; MaximizeBox = false; Text = "Recognizer Performance"; ResumeLayout(false); } } Memory and Storage of Ink
In addition to speed, another key consideration of any performance analysis is the problem of memory and storage requirements. When it comes to memory, the Tablet PC Platform can be surprisingly demanding. This makes a clear understanding of digital ink s memory requirements imperative to making design decisions regarding your application. Fortunately, the platform can be efficient in how it stores ink. We will look at and compare the size requirements of various storage formats for ink.
First, let s take a look at digital ink s memory usage. The MemoryPerf application allows some basic investigation of memory consumption. It loads an ISF file and subsequently replicates the strokes in the file quite a few times. These replications are meant to produce a significant number of strokes in total, in an effort to asymptotically minimize the amount of non-ink-related memory overhead. In particular, making sure the strokes themselves compose the bulk of consumed memory should reduce the effects of platform (and application) memory overhead not directly related to the ink strokes. After replicating the strokes, MemoryPerf gives the user a chance to take another memory measurement. The difference between the pre-replication memory and the post-replication memory is thus a fair estimate of the amount of memory actually used.
MemoryPerf does not provide its own method of measuring memory because there is no simple piece of code that would yield a reasonable measurement. Instead, you should use another tool to gauge the amount of memory consumed. One such tool is the Windows Task Manager, which, though not always strictly accurate, offers a good first-order approximation of the memory used by any application.
NOTE
You might be tempted to use GC.GetTotalMemory, a compelling function provided by the Microsoft .NET garbage collector. However, in the case of the Tablet PC Platform, any figure obtained from GC.GetTotalMemory would be grossly understating the actual memory usage because it tracks only memory allocated in managed code (using the .NET Framework). The Tablet PC Platform managed API was largely implemented as a set of .NET-compatible wrappers around classic C/C++ code, whose memory usage is not tracked by the .NET garbage collector.
The code for MemoryPerf is listed here and is also available in the companion content.
MemoryPerf.cs
//////////////////////////////////////////////////////////////////// // MemoryPerf.cs // // (c) 2002 Microsoft Press, by Philip Su // // This program measures memory performance //////////////////////////////////////////////////////////////////// using System; using System.IO; using System.Drawing; using Microsoft.Ink; class MemoryPerf { public static void Main(string[] args) { Ink ink; if (args.Length == 0) { Console.WriteLine("Usage: MemoryPerf.exe [file.ISF]"); return; } LoadInk(args[0], out ink); if (ink != null) { RunTest(ink); } } private static void RunTest(Ink ink) { const int cCopies = 150; // Copies of ink to make Strokes strokes = ink.Strokes; Rectangle rc = strokes.GetBoundingBox(); int cStrokes = strokes.Count; int cPoints = 0; foreach (Stroke s in strokes) { cPoints += s.PacketCount; } Console.WriteLine("(measure memory and press return)"); Console.ReadLine(); for (int i = 0; i < cCopies; i++) { ink.AddStrokesAtRectangle(strokes, rc); } Console.Write("{0} strokes and {1} points total ", cStrokes * cCopies, cPoints * cCopies); Console.WriteLine("(measure memory and press return)"); Console.ReadLine(); } private static void LoadInk(string strFile, out Ink ink) { ink = null; try { FileStream fs; int cbRead = 0; // Count of bytes read byte[] rgb; // Our ISF bytes from disk // Attempt to load the ISF file. Make sure that // this file is in the directory you run this // application from! fs = new FileStream(strFile, FileMode.Open); rgb = new byte[fs.Length]; cbRead = fs.Read(rgb, 0, rgb.Length); // Read ISF if (cbRead == fs.Length) { ink = new Ink(); ink.Load(rgb); // Load the ISF into ink } fs.Close(); } catch (FileNotFoundException) { Console.WriteLine(strFile + " not found!"); } } } Table 9-1 presents a heavy usage scenario involving a large set of plain strokes.
Total Strokes | 32,850 |
Total Packets (Points) | 671,400 |
Average Packets per Stroke | 46 |
Packet Size | 12 bytes |
Total Memory Used | 20 Mb |
Average Memory per Stroke | 620 bytes |
Average Memory per Packet | 31 bytes |
First, it s important to recognize that 32,000+ strokes of ink equate to about 68 pages of wide-ruled paper filled with cursive writing (or about 18 pages of print). This might be a lot of ink compared to scenarios you re thinking about. However, the critical figure in the table is the average memory per stroke, which is about 620 bytes in this scenario. Using this figure, you can arrive at some rough estimates about the memory requirements of your ink-enabled application (a page of cursive might be about 300 Kb, for example).
However, the average memory per stroke is heavily dependent on two things: the average stroke packet count and the packet size. Both of these variables can vary quite dramatically depending on the situation. The stroke packet count varies with the amount of time the pen is down as well as the sampling rate of the hardware being used. Typical Tablet PC hardware might capture 120+ points per second or beyond. Assuming that a typical cursive stroke takes on the order of two seconds, the average stroke packet count you deal with might be quite higher than that in Table 9-1. The other factor that might further inflate these figures is the packet size you choose. By requesting more information per packet (for example, pressure or tilt), each packet s memory requirements go up. If you want each packet to contain a lot of information, be prepared for a lot of memory to be used. When you compound average stroke packet count and packet size, it can cause a serious memory usage concern, so investigate your ink feature needs a bit before finalizing a design.
You might be wondering why the average memory per packet is almost triple the packet size. This is likely due to per-stroke implementation overheads of the platform, but there is no concrete evidence. What matters most, though, is not what the platform says the packet size is instead, the empirically measured memory consumption is all that matters when it comes to estimating how much memory your application will need.
We would be remiss in our discussion of memory requirements not to reiterate the implications of undo. If your application supports undo, users will expect to be able to undo ink modifications as well. However, as we discussed earlier in the chapter, the Tablet PC Platform SDK provides no native support for undoing modifications. In fact, the design of the managed API makes it downright hard in certain cases to implement undo without cloning each Ink object as it s modified. If you implement undo by cloning, the memory requirements of your application might rise dramatically as the amount of ink being worked with increases. Imagine that you have a page of writing housed in one Ink object, which might use somewhere on the order of several hundred kilobytes of memory. As the user enters new ink strokes, you might have to clone the Ink object with each additional stroke in order to support undoing of the strokes. This process quickly becomes unwieldy if you support multilevel undo and begs for a smarter solution. Depending on whether and how you plan to support undo, you might need to design around this fundamental limitation of the platform by either partitioning ink into smaller Ink objects or by implementing a smarter undo. You can investigate the undo issue further by experimenting with the InkPadJunior sample application presented later in this chapter it implements multilevel undo by cloning Ink objects as they change.
NOTE
If you monitor memory usage as you enter more and more ink in the application, you will witness the unfortunate O(n2) memory consumption of such an implementation. For each ink stroke added, all previous ink strokes must be cloned; therefore, given the nth ink stroke, each previous stroke must be cloned n-1 times.
The good news is that the size of digital ink is much smaller when persisted than when it s in memory. A typical storage requirement breakdown is presented in Table 9-2, followed by the source code used to generate the data. The source code is simply a different implementation of the RunTest function in the MemoryPerf application. The code for the entire application is available as the StoragePerf solution in the companion content.
Total Strokes | 1344 |
Total Packets (Points) | 62,550 |
Average Packets per Stroke | 46 |
Packet Size | 12 bytes |
No Compression ISF | 277 Kb (211 bytes per stroke / 4.5 bytes per packet) |
Default Compression ISF | 156 Kb (119 bytes per stroke / 2.5 bytes per packet / 56% of uncompressed) |
Maximum Compression ISF | 156 Kb (119 bytes per stroke / 2.5 bytes per packet / 56% of uncompressed) |
GIF (Default ISF) | 165 Kb (105% of default ISF) |
Base64 (Default ISF) | 208 Kb (133% of default ISF) |
private static void RunTest(Ink ink) { int cStrokes = ink.Strokes.Count; int cPoints = 0; int cbNone = 0; int cbNormal = 0; int cbMaximum = 0; int cbGIF = 0; int cbBase64 = 0; byte[] rgb; foreach (Stroke s in ink.Strokes) { cPoints += s.PacketCount; } Console.WriteLine("{0} strokes and {1} points total:", cStrokes, cPoints); rgb = ink.Save(PersistenceFormat.InkSerializedFormat, CompressionMode.NoCompression); cbNone = rgb.Length; rgb = ink.Save(PersistenceFormat.InkSerializedFormat, CompressionMode.Default); cbNormal = rgb.Length; rgb = ink.Save(PersistenceFormat.InkSerializedFormat, CompressionMode.Maximum); cbMaximum = rgb.Length; rgb = ink.Save(PersistenceFormat.Gif); cbGIF = rgb.Length; rgb = ink.Save(PersistenceFormat.Base64InkSerializedFormat); cbBase64 = rgb.Length; Console.WriteLine("No compression: {0} bytes", cbNone); Console.WriteLine("Default compression: {0} bytes " + "({1}% of no compression size)", cbNormal, cbNormal * 100 / cbNone); Console.WriteLine("Maximum compression: {0} bytes " + "({1}% of no compression size)", cbMaximum, cbMaximum * 100 / cbNone); Console.WriteLine("GIF (default ISF): {0} bytes " + "({1}% of default ISF)", cbGIF, cbGIF * 100 / cbNormal); Console.WriteLine("Base64 (default ISF): {0} bytes " + "({1}% of default ISF)", cbBase64, cbBase64 * 100 / cbNormal); } Using our handy 30-line, 10-words-per-line approximation of a page, we see that the data in Table 9-2 pertains to about 4 5 pages of ink. Under default compression, each page of ink should take about 36 Kb of storage, which is not bad at all. Remember that the amount of data per packet will affect results.
Most of the storage performance data is self-explanatory, with a few notable anomalies that we ll discuss. First, notice that even without compression, we needed only 4.5 bytes on average to store each 12-byte packet. This is surprising indeed! The only explanation we ve been able to surmise is that perhaps the platform removes duplicate points (say, if the pen didn t move between two samples) even in no compression mode. In any case, this result is startling. Another exception that we need to bring up is that the overhead for the GIF output format does not seem to grow with the number of strokes or packets stored. This makes some sense because the only overhead that should be incurred is the GIF-related headers and tags. Another consequence of this observation is that even the smallest ink object will take several kilobytes (upwards of 10) to store in the GIF format due to this overhead. Finally, Base64 s 33 percent bloat is exactly as expected because the Base64 algorithm turns every 3 binary bytes into 4 characters.
EAN: 2147483647
Pages: 73