Using Gestures to Control Tablet Media Player

 Download CD Content

Overview

If you recall from Chapter 16, Creating an MP3 Player, we created an MP3 player that allows the user to write out a command that is recognized and a certain behavior is executed, such as playing or stopping a file. Although it obviously works, it is definitely not an easy solution from a user standpoint, as the need to write the word 'play' to play a file is very tedious. In this chapter, we look at a much better way to control these types of applications by using gestures to control a media player (see Figure 18.1).

Figure 18.1: Built-in gestures from Table 18.1, viewed from left to right.

Note 

The source code for the projects are located on the CD-ROM in the PROJECTS folder. You can either type them in as you go or you can copy the projects from the CD-ROM to your hard drive for editing.

Tablet Gestures

If you have ever used a PalmPilot, you have seen the way a device can take advantage of a type of shorthand called graffiti. The Palm works by allowing the user to write a simple version of complex characters-making entering text an easier objective. The Tablet PC provides a much deeper implementation of a shorthand system known as gestures.

The gestures are available as part of the standard Tablet PC Platform SDK. When you develop an application, the gestures are not enabled by default. Rather, if you want to enable one of the built-in gestures, you must instruct the application that you are doing so. Most gestures have a 'hot point,' which is a distinguishing point in the geometry of the gesture. The hot point can be used to determine where the gesture was performed. The gestures APIs provide the HotPoint property of the Gesture event, making it possible to determine the hot point for a given gesture. Not all gestures have a specific hot point; for these gestures, the starting point is reported as the hot point. Using both Figure 18.1 and Table 18.1, you can see an example of the gestures and the location of the hot point.

Table 18.1: Gestures

Name	Hot Point	Notes
Scratch-out	Starting point	Make minimum three strokes close together
Triangle	Starting point	Don't lift pen while drawing
Square	Starting point	Start at upper-left corner
Star	Starting point	Draw five points
Check	Corner	Draw upward portion 2-4 times larger
Curlicue	Starting point	Angled from lower left to upper right
Double-Curlicue	Starting point	Same as previous but twice
Circle	Starting point	Single stroke starting at top
Double-circle	Starting point	Draw circles overlapping
Left-semicircle	Starting point	Draw from right to left
Right-semicircle	Starting point	Draw from left to right
Caret	Apex	Both sides equal length
Inverted-Caret	Apex	Same as previous but inverted
Chevron-left	Apex	Both sides equal length
Chevron-right	Apex	Both sides equal length
Arrow-up	Arrow head	Limit to two strokes
Arrow-down	Arrow head	Limit to two strokes
Arrow-left	Arrow head	Limit to two strokes
Arrow-right	Arrow head	Limit to two strokes
Up	Starting point	Single upward line (draw fast)
Down	Starting point	Single downward line (draw fast)
Left	Starting point	Single line to left (draw fast)
Right	Starting point	Single line to right (draw fast)
Up-left	Point of change	Single stroke starting with up stroke
Up-right	Point of change	Single stroke starting with up stroke
Down-left	Point of change	Single stroke starting with down stroke
Left-up	Point of change	Single stroke starting with left stroke
Left-down	Point of change	Single stroke starting with left stroke
Right-up	Point of change	Single stroke starting with right stroke
Right-down	Point of change	Single stroke starting with right stroke
Up-down	Point of change	Start with up drawing close together
Down-up	Point of change	Start with down drawing close together
Left-right	Point of change	Start with left drawing close together
Right-left	Point of change	Start with right drawing close together
Up-left-long	Point of change	Start up with left 2x longer
Up-right-long	Point of change	Start up with right 2x longer
Down-left-long	Point of change	Start down with left 2x longer
Down-right-long	Point of change	Start up with right 2x longer
Exclamation	Center of line	Draw dot soon after line
Tap	Starting point	Perform quickly
Double-tap	Starting point	Perform quickly

Building the Media Player

Microsoft provides the Media Player ActiveX control as a simple way to offer media player capabilities. We are going to take advantage of this control as we build an application that can open various movie files and that is controlled by gestures.

To begin this application, create a new Windows Forms application and add an OpenFileDialog to it. You can leave its name as the default given. In the middle of the form, add an InkPicture control, and in the upper right of the form, add a separate one (should look like Figure 18.2) leaving their names as InkPicture1 and InkPicture2.

Figure 18.2: Position the controls on the form.

Now, let's add the media player control to the form. Choose Project | Add Reference to display the Add Reference dialog box. Next, choose the COM tab and then locate the Windows Media Player OCX control. You can position the media player control on the form as seen in Figure 18.3. This control should approximately cover the InkPlayer control we placed on the form earlier. We're going to use the InkPlayer control to allow us to annotate on the movie.

Figure 18.3: The Media Player OCX positioned on the form.

The next step for this project is to add a Label control and position it above the InkPicture2 control. Change its Text property to read 'Gesture'. The final thing we need is a StatusBar control added to the form. When it's added, it automatically positions itself along the bottom of the form, which is where the status bar is normally placed.

Writing Some Code

Before we get to most of the code, we need to add a reference to the Microsoft Ink API. Therefore, we need to add the following line to the top of the code:

Imports Microsoft.Ink

We'll use the Form_Load event to set several properties very important to our application. First, set the InkPicture1 control to have a transparent background color, remove its border, and then bring it to the front so that it is available for annotating over the video. Here are the three lines of code:

InkPicture1.BackColor = Color.Transparent
InkPicture1.BorderStyle = BorderStyle.None
InkPicture1.BringToFront()

Now, let's turn our attention to the other InkPicture control. This one will be used to handle the gestures for our application, which will control the features such as opening, playing, and pausing a video. First, we'll set its borderstyle and backcolor so that it separates itself from the rest of the interface and is very visible to the user. As we are interested in using the control to capture gestures only, we are going to set its Collection mode to GestureOnly, and then we'll set the gestures that it will try to recognize. Our application is going to use the following gestures:

Square: Stop

UpDown: Pause

ChevronRight: Play

ArrowRight: Volume increase

ArrowLeft: Volume decrease

Circle: Open

Here is the code:

InkPicture2.BorderStyle = BorderStyle.Fixed3D
InkPicture2.BackColor = Color.White
InkPicture2.CollectionMode = CollectionMode.GestureOnly
InkPicture2.SetGestureStatus(ApplicationGesture.Square, True)
InkPicture2.SetGestureStatus(ApplicationGesture.UpDown, True)
InkPicture2.SetGestureStatus(ApplicationGesture.ChevronRight, True)
InkPicture2.SetGestureStatus(ApplicationGesture.ArrowRight, True)
InkPicture2.SetGestureStatus(ApplicationGesture.ArrowLeft, True)
InkPicture2.SetGestureStatus(ApplicationGesture.Circle, True)

The last two things we need to handle are the status bar's Text property, which we set to 'No file loaded.', and the filter for the open dialog box so that we can only open MPEG and AVI videos:

OpenFileDialog1.Filter = "MPEG Video (*.mpg)|*.mpg|Video for Windows (*.avi)|*.avi|All Files (*.*)|*.*"
StatusBar1.Text = "No file loaded."

Now that we have our gestures, we can use the InkPicture2_Gesture event to determine which gesture was written and then perform the appropriate actions. We'll use an If...Then statement at the beginning of the procedure to check the current gesture's confidence. We can test to see if the recognition confidence is strong, and if so, we will use a Case statement to determine which type of gesture was written and instruct the AxMediaPlayer control appropriately. If the gesture confidence is less than strong, we will exit without performing any activities with the AxMediaPlayer. Before we exit, we set the status bar's Text property to 'Command Not Recognized' so that the user realizes that the command they attempted did not perform any function.

We have already discussed the gestures and what each of them will do, so here is the code:

If e.Gestures(0).Confidence = RecognitionConfidence.Strong Then
 Select Case e.Gestures(0).Id
 Case ApplicationGesture.Square
 AxMediaPlayer1.Stop()
 StatusBar1.Text = "Stop"
 Case ApplicationGesture.ChevronRight
 AxMediaPlayer1.Play()
 StatusBar1.Text = "Play"
 Case ApplicationGesture.UpDown
 AxMediaPlayer1.Pause()
 StatusBar1.Text = "Pause"
 Case ApplicationGesture.ArrowLeft
 AxMediaPlayer1.Volume = AxMediaPlayer1.Volume - 1
 StatusBar1.Text = "<- Vol = " & AxMediaPlayer1.Volume.ToString
 Case ApplicationGesture.Circle
 OpenFileDialog1.ShowDialog()
 AxMediaPlayer1.Open(OpenFileDialog1.FileName)
 Case ApplicationGesture.ArrowRight
 AxMediaPlayer1.Volume = AxMediaPlayer1.Volume + 1
 StatusBar1.Text = "-> Vol = " & AxMediaPlayer1.Volume.ToString
 StatusBar1.Text = AxMediaPlayer1.FileName
 End Select
Else
 StatusBar1.Text = "Command Not Recognized"
End If

The application is functional at this time, but we have a few additional things we need to handle. First, we need to change the drawing color for both InkPicture controls, depending on which control the pen is entering. We'll use both MouseEnter events for this aspect of the application:

Private Sub InkPicture2_MouseEnter(ByVal sender As Object, ByVal e As System.EventArgs) Handles InkPicture2.MouseEnter
 InkPicture2.DefaultDrawingAttributes.Color = Color.Red
End Sub

Private Sub InkPicture1_MouseEnter(ByVal sender As Object, ByVal e As System.EventArgs) Handles InkPicture1.MouseEnter
 InkPicture1.DefaultDrawingAttributes.Color = Color.Blue
End Sub

The final thing we need to add to the application is a way to pause the AxMediaPlayer currently playing file so that we can annotate a given frame. We can use the MouseDown event procedure for InkPicture1 to test the current filename. If there is a filename, we then test to see if the media player is currently playing. If it is, we pause the player so that we can proceed to annotate it. Lastly, we set the status bar's Text property to 'Paused for annotation.'

Here is the code:

Private Sub InkPicture1_MouseDown(ByVal sender As Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles InkPicture1.MouseDown
 If axMediaPlayer1.FileName "" and AxMediaPlayer1.PlayState = MPPlayStateConstants.mpPlaying Then
 AxMediaPlayer1.Pause()
 StatusBar1.Text = "Paused for annotation."
 End If
End Sub

You can test the application to see if it works. Figure 18.4 displays an open gesture with an OpenFileDialog and Figure 18.5 shows a video being annotated.

Figure 18.4: A file being opened.

Figure 18.5: A video file being annotated.

Summary

In this chapter, we created a video player that was controlled using gestures. The application also used an additional InkPicture control to allow us to annotate over the content of the video. In Chapter 19, Getting Started with Microsoft Agent, we build our first speech-enabled application.