Manual Testing in Team System

Manual tests are first-class citizens in Team System — you can run automated tests and manual tests side by side. You can track the progress of a test in the Test Results window, use import/export functions, and drill down into any tests at any time. You can also generate metrics and reports derived from manual tests.

To use the manual testing features of Team System, you should install and use the Team Edition for Software Testers or Team Suite version of the product. Microsoft Word is not necessarily required to create manual tests (you can create plaintext-based manual tests), but overall, it provides a better experience by allowing you to flexibly format the content.

Traditionally, manual testing has been a "paper-based" process. You can also find third-party applications that can manage manual tests. The main advantage Team System offers is integration with Visual Studio 2005 and Team Foundation Server. Team System provides a framework in which the software test engineer can focus on test planning and generating requirements, rather than having to worry about learning a new toolset because the tester is primarily working with Word or text documents.

Creating manual tests

Creating a manual test in Team System is quite easy. Simply open your target project, right-click on the project name in the Solutions Explorer and select Add Test Project. You can also add a project by clicking File New New Project. Then you must select the Test Project type. Figure 16-1 shows a screenshot of the New Project dialog box.

Figure 16-1

Once you have created the test project, you can add new manual tests by right-clicking on the test project within the Solutions Explorer and selecting Add New Test. This option to add a new test is also accessible from the Test menu within the Visual Studio 2005 IDE. The manual test will automatically appear within your test project in the Solutions Explorer.

Note

As a shortcut, you can also add a manual test (Word format) directly, without going through the wizard, by right-clicking on your project in the Solution Explorer and selecting Add Manual Test (Word Format).

Figure 16-2 shows the Add New Test Wizard. Clicking on any of the tests in the Templates pane will give you a more detailed explanation in the Description pane on the right.

Figure 16-2

Configuring manual test options

When you create a test project, two tests are added by default: a unit test and a Word-based manual test. You can modify your default test project to include manual tests by clicking Tools Options Test Tools Test Project. The test project Options window is shown in Figure 16-3.

Figure 16-3

To add the manual test file to the default test project, simply check the Manual Test (text format) check- box and click the OK button. You can set individual test project settings for any development language, including Visual C#, Visual Basic, or Visual C++.

You can also control the behavior of manual tests by accessing the Default Dialog Box Action options (see Figure 16-4). To access the options, simply click Test Options Test Tools Default Dialog Box Action.

Figure 16-4

Some of the manual testing options you can configure include the following:

• When opening manual tests in Word format when Microsoft Word 2003 (or later) is not installed: Team System requires Microsoft Word 2003 because it embeds special metadata inside the document to identify it as a test (see the section entitled "Creating a Visual Studio 2005 test template" for more details). The two options you have at your disposal include "Always prompt" and "Always view the manual test in read-only mode."

• When changing a property of a manual test in the Properties window while the manual test is open in Microsoft Word: Because Microsoft Word is an external program, any changes to the properties window within Visual Studio will not automatically propagate to your Word document when it is open. To get around this problem, you can set one of two options in the Default Dialog Box Action window: "Always prompt" or "Always save the manual test."

• When submitting a test run that contains manual tests: Your options are "Always prompt" or "Don't prompt." Because manual tests require the intervention of a tester, you will definitely want to set this option to "Don't prompt" if you are planning to deploy manual tests in a test run on the build server. By setting "Don't prompt," Team Foundation Build will skip over the manual tests and continue with the automated tests. Otherwise, you will get the prompt shown in Figure 16-5.

  
  Figure 16-5

• When a remote test run contains manual tests: You have three options: "Always prompt," "Always remove manual tests from test run and continue," or "Always abort the test run." These options are quite important if you want to automate the tests on your build server. Otherwise, you will either receive a prompt that will stop the automated tests dead in its tracks or the test run will be aborted. In most cases, you will want to select "Always remove manual tests from test run and continue."

• When a manual test result window is to be opened for execution: When you include manual tests in your test project, Visual Studio will pop up a series of dialog boxes to prompt the tester. You can choose to disable some of these prompts to streamline the testing process. The two options you can set include "Always prompt and disable the manual test result window" or "Always directly disable the manual test result window."

• When working on a manual test and it is aborted: If a manual test is aborted for any reason whatsoever, you can choose whether to allow Visual Studio to prompt the tester to enter a test result or skip over the test completely. The options for this include "Always prompt and disable the manual test result window" or "Always directly disable the manual test result window."

• When closing a running manual test: If you close a manual test mid-run, you can set whether the test will abort or whether the tester can enter a test result before shutting it down. The two options for this scenario are "Always prompt" or "Always abort the test."

Manual test templates

Team System provides two default manual test templates: a Microsoft Word template and a Text template. These default tests can be found as part of the Team Suite and Team Tester SKUs. All of the test templates are located in the following program directory:

      Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\<Language=\1033\ 

The default manual tests are available for a variety of programming languages, including Visual C#, Visual Basic.NET, Visual J#, and Visual C++. The manual test templates are located in two .zip files:

• ManualTestWordFormat.zip

• ManualTestTextFormat.zip

Each of these .zip files contains two files:

• ManualTest.mtx (or ManualTest.mht): The actual test template files

• ManualTest.vstemplate: Used to create project items, which can be consumed by Visual Studio 2005 and shown in the Add New Test Wizard (see Figure 16-2)

Note

Only Microsoft Word and text-based manual tests are supported in the current version of Team System. Other file types (such as Excel) are not supported.

Creating custom manual test templates

There are several ways to create custom manual test templates: add a modified manual test in your project, export your manual test template as a new item, or package up a new manual test template for the Visual Studio 2005 IDE. Please note that the third option is the most complex and prone to errors.

Adding a custom manual test in your project

You can add a custom manual test in your project by following these steps:

1. Create a new test project.

2. Take the default manual test template and create a copy of it.

3. Add the copy to your project and launch the Word or text file to edit the contents of the manual test.

4. Once you are done, rename the file (give it a filename such as CustomManualTest.mht) and save it. You can then copy and reuse the new template as you like.

Exporting your custom manual test as a new item

You can export a custom manual test as an item template. That way, users will be able to import it using the Add New Item dialog box. Here are the steps to create a new item template:

1. Open your test project. Make the modifications you want to your test.

2. Click File Export Template.

3. The Export Template Wizard will appear, as shown in Figure 16-6. Select the Item template option and the appropriate test project.

   
   Figure 16-6

4. Select the manual test(s) you would like to export (see Figure 16-7).

   
   Figure 16-7

5. You will then arrive at the Select Item References page (as shown in Figure 16-8). You don't have to select any of the options because manual tests don't require special assembly references to run. Click Next.

   
   Figure 16-8

6. During the final step, you can select an icon, a name, and description for your project (shown in Figure 16-9). If you check "Automatically import the template in Visual Studio," the template will be made available locally every time you launch the IDE. Checking the "Display an explorer window on the output files folder" option will show you all your templates (including the one you just created) in the following directory: C:\Documents and Settings\…\My Documents\Visual Studio 2005\My Exported Templates.

Figure 16-9

Creating a Visual Studio 2005 test template

Of all the methods outlined so far, this is by far the trickiest. The advantage of creating a Visual Studio 2005 template is that it is conveniently added with the other templates in the Add New Test dialog.

1. Copy ManualTestWordFormat.zip on your desktop and extract ManualTest.mht. Customize it to your liking using Microsoft Word. Assuming you installed Visual Studio 2005 on your C: drive, the path to the zip file should be C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\<language=\1033\ManualTestWordFormat.zip.

2. Please note that TestID and TestType are required custom Word properties that enable your document to be recognized as a manual test within the IDE. To manually set the CustomDocumentProperties, edit the source of the .mht file (using a text editor other than Word) and add the values highlighted below.

   The TestType value should always remain Manual Test. Otherwise, Visual Studio may not recognize your document as part of a manual test. The TestID Globally Unique Identifier (GUID) should be left at the value of $guid1$. Visual Studio will automatically populate this field with a new GUID once a test is created within the IDE, to uniquely identify each manual test in the test results.

        <html xmlns:o="urn:schemas-microsoft-com:office:office"     xmlns:w="urn:schemas-microsoft-com:office:word"     xmlns:dt="uuid:"     xmlns="http://www.w3.org/TR/REC-html40"=     <head=     <meta http-equiv=Content-Type content="text/html; charset=us-ascii"=     <meta name=ProgId content=Word.Document=     <meta name=Generator content="Microsoft Word 11"=     <meta name=Originator content="Microsoft Word 11"=     <link rel=File-List href="ManualTest1_files/filelist.xml"=     <title=To do: set description</title=     <!--[if gte mso 9]=<xml=      <o:CustomDocumentProperties=      <o:TestType dt:dt="string"= Manual Test</o:TestType=     <o:TestID dt:dt="string"=$guid1$</o:TestID=     </o:CustomDocumentProperties=     </xml=     ...     </head= 

3. Once you are done, replace the .mht file in ManualTestWordFormat.zip with the one you just edited. Rename ManualTestWordFormat.zip to a filename of your choice (such as CustomManualTest.zip) and paste it into the following directory: C:\ProgramFiles\ Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\<language=\1033\.

Integrating your custom manual test template into Visual Studio 2005

Once the custom test has been added in the correct directory, you can refresh the ItemTemplatesCache directory using the following command:

     devenv /setup 

Team System and Visual Studio 2005 doesn't have a mechanism to enable you to deploy manual tests to multiple users. Here are a few solutions to get around this problem:

• Custom MSI: You can create an installer that will place the template in the right directory. You can then silently install them on any of your team members' computers using Group Policies and a solution such as Systems Management Server. See the section entitled "Distribute Visual Studio Templates and Policy Wizards in a Deployment Project" in the MSDN Documentation for more information. For information on how to deploy files in your infrastructure, visit the Microsoft TechNet website at http://www.technet.microsoft.com.

• Project Portal: You can add the .zip file in your custom process and request that your team members install them manually.

• Source Control: You can both check the files into source control and request that your team members use them, or map the source control system with the Visual Studio Templates directory. Once your team members retrieve the latest source code updates, the templates will also be updated.

Opening test templates using the Open With wizard

To open a test template using the Open With Wizard, simply right-click on the file in the Solutions Explorer and select Open With. Figure 16-10 shows a screenshot of the Open With Wizard (listing an assortment of editors you can pick from to manipulate your manual tests).

Figure 16-10

Note

If you choose any editor other than the Manual Test Editor (Default), your test may be rendered as HTML tags, rather than an editable test template.

Microsoft Word format template

The Microsoft Word version of the manual test template has an .mht file extension (which denotes that it is a Microsoft HTML document). If you have Microsoft Word installed on your system, you should select this template type because it will enable you to nicely format your test.

You can also embed screenshots, photos, and other rich content. Figure 16-11 illustrates the layout of a Word-based manual test template.

Figure 16-11

Text Format template

The Text Format template has an .mtx extension (a Microsoft Text document — not to be confused with the Twain device driver). The Text Format template is a great deal simpler than its Microsoft Word counterpart. On the downside, you can't embed any rich content such as pictures. On the upside, it doesn't require Microsoft Word to be viewed and edited. Figure 16-12 shows a screenshot of a Text Format Manual Test template displayed in the Manual Test Editor in Visual Studio 2005.

Figure 16-12

Manual test structure

All manual test templates are structured the same way. There are five main sections in a manual test:

• Test Title

• Test Details

• Test Target

• Test Steps

• Revision History

The Test Title and Test Details sections describe what kinds of tests are being performed on your application. This is an important step because the test will have to be reproduced in the future, so be detailed in your wording. You can only enter two to three words in the title.

The Test Target section describes the specifics of the functionality covered by your test. The Test Steps section describes in detail what steps are needed to accomplish the test. Each step is numbered with a description of the step and the outcome. It is crucial that the test be sufficiently documented in order to be reproducible. In addition, try to minimize the number of steps to reproduce.

Finally, the Revision History section tracks when the test was revised and by whom. You can enter the author's name (or e-mail alias), a description of the revisions made to the test, and the current date and time. The data entered on this template is handled and measured the same as any other test in Team System, using the same underlying framework.

Don't feel you have to stick to these five elements. You can expand on it, add other sections, and remove sections. Team System does not parse the contents of your manual test document, so you have a lot of flexibility in the way you structure your content in the document. In fact, grouping manual tests can be as simple as cutting and pasting several tests into one document.