Creating and Running a Generic Test

In the following sections, you will create a new generic test, see how it runs, and view the results of the test. In the process, we will discuss exit codes (otherwise known as ErrorLevels).

Creating a generic test

For this example, let's assume that you have a simple C# app that runs a web search engine query. You have half a dozen unit tests to test your code, but before they kick off you want to verify that the server exists. One possible solution would be to write a unit test with code that tries to hit the server. Another possible solution is shown in the following example, which uses the ping.exe utility included with Windows.

To begin, start Visual Studio Team System. Select File image from book New image from book Project. This will bring up the New Project dialog. Select the Test Documents under the Test Projects type and the Test Project template, give it a name, and click OK (see Figure 17-2).

image from book
Figure 17-2

A new test project will be created with a text file of notes (AuthoringTests.txt), a manual test (ManualTest1.mht), and a unit test (UnitTest1.cs). These can all be deleted from the project for the sake of this example.

This is where we use a generic test. Right-click on the project name to bring up the context menu and select Add image from book New Test. Select New Test instead of Generic Test so that you can give the test a name using the Add New Test dialog shown in Figure 17-3.

image from book
Figure 17-3


If you select Generic Test from the context menu, a file will be immediately added to the project, but giving the file a meaningful name is a more involved process. First the file would be renamed (by right- clicking the file and choosing Rename or pressing F4), but this would not affect the actual displayed verbose name of the test. You would need to view the Test Manager window, select the new test, view the Properties window, and then edit the Test Name property. All this is done for you using the Add New Test dialog, so the process is not shown here.

Name the file Ping.GenericTest and click OK. As you can see, a generic test is just one of the several built-in test types available. Other types include load test, manual test, ordered test, unit test, and web test and are discussed in other chapters of this book.

Next, the generic test editor will appear (see Figure 17-4). This editor contains the main properties of the generic test. Other properties can be accessed using the Property window, as described in the section "Running the generic test."

image from book
Figure 17-4

In the text box at the top of the editor under the "Specify an existing program to wrap as a generic test:" label, enter the location and name of the program. In our case, C:\WINDOWS\system32\ping.exe. The option of selecting a program by clicking on the ellipsis located next to the text box is also available. Under Run settings, in the "Command line arguments" text box, enter -n 1 Now save your entries (Ctrl+S) and close the editor for this generic test (Ctrl+F4).

The –n 1 tells ping.exe to attempt to reach the server only once. ping.exe sets an ErrorLevel of 0 for success (Passed) and 1 for failure (Failed), which is what the generic test type expects.

The filename of the generic test does not affect the displayed name of the test. To rename the test, view the Test Manager by selecting Test image from book Windows image from book Test Manager.

Figure 17-4 also shows a number of options that you can configure in the generic test properties dialog box:

  • Specify an existing program to wrap as a generic test: The first text box is the full path to the executable to be run. When the generic test is run, the executable will be run one time. The test engine will use the executable's output (ErrorLevel or XML) to determine whether the test was successful.

    This file must be either a Win32 Windows (GUI) or a Win32 Console application. Scripts, such as JScript or VBScript, are executed by cscript.exe, a Win32 console application. It cannot be a DOS command or a 16-bit app, because the test engine tracks the state of the thread during execution and captures the resulting ErrorLevel, which requires a Win32 proc.


    To run a batch file, use the syntax C:\windows\System\cmd.exe/c[batchfile].

  • Command-line arguments to pass to the generic test: The second text box (the first under the Execution Settings area) is used to provide additional arguments to be passed to the executing app. As with all command-line arguments, filenames with spaces anywhere in the path must be contained in quotes — for example: "C:\Program Files\Internet Explorer\iexplorer.exe."

  • Additional files to be deployed with this generic test: You can select files that will be deployed along with the tests. These files will be copied to the directory stored in the testdeploymentdir environment variable. These are usually files that your code under test requires to run properly.

  • Environment Variables: You can add additional environment variables to the process in which the external application is run. These variables will only be available to the application when it is started and run by the generic test. The following table describes the additional available environment variables that the test engine sets.


In the example values in the following table, NOAHC refers to the name of the computer on which the test is being executed; MySolution is the directory containing the solution file; and find guts.generictest is the name of the currently running test.




Directory to which this test generic test was deployed. Example value: C:\MySolution\TestResults\Noahc_NOAHC 9_2_2005 2_28_20 AM\Out\find guts.generictest


Directory that contains the original source .GenericTest file.


Directory that will contain log files, code coverage data, etc.


Directory to which the files the user selected to be deployed are copied. Example value: C:\MySolution\TestResults\ Noahc_NOAHC 9_2_2005 2_28_20 AM\Out\find guts.generictest


Directory to which any output from the application should be copied so that it is picked up by the test engine and made available in the reports. Example value: C:\MySolution\TestResults\ Noahc_NOAHC 9_2_2005 2_28_20 AM\In\NOAHC\

If you created a .NET executable being run by the generic test, you can obtain the value of the environment variables in your code using the Environment class:

      string dir = Environment.GetEnvironmentVariable("testlocation"); 

  • Redirect standard output and error to test result: If checked, the test engine captures the process's output and error streams and saves them as part of the test run's report.

  • Display execution application (window or console) at runtime: When checked, this will display the app to the user. The preceding checkbox must be unchecked if the user is to see any output on the console.

  • Exit test run if run duration (in milliseconds) exceeds: The test will terminate with a Failed result if the time elapses. If this is unchecked, the test run may be permanently frozen until the test run is manually shut down.

Back in the Microsoft DOS days, an executing application could leave a trail when it exited by returning a single integer back to the operating system as an exit code. This exit code was called an ErrorLevel because it was designed to report an error, and, if so, a value indicating the type of error. In batch files, this ErrorLevel could then be used to test the result of an execution, and branch if there was an error. An ErrorLevel of 0 indicated a success, and anything else was some form of an error. The DOS batch if errorlevel [x] statement would result to true if the last statement's ErrorLevel was equal to or greater than [x]. For example:

     @echo off     rem QuickTest.bat     ping -n 1     if errorlevel 1 goto problem     echo Success!     goto end     :problem     echo There was a problem, batch aborted.     echo ErrorLevel: %errorlevel%     goto end     :end 

The environment variable errorlevel is set by the command shell to the exit code of the application. This would branch on any ErrorLevel greater than 0. The only common standard is that 0 indicates a success and anything else is a failure. The ErrorLevel is a standard integer with possible negative and positive values. The interpretation of what type of error occurred within the application by the resulting ErrorLevel is determined by the application. One would have to read the application's documentation, view its code, or deduce by trial and error what exit code was produced by what error.

You can easily return an exit code from a .NET console application. Most importantly, change the Main method from returning void to returning an int. The following short example will return the exit code provided on the command line. Its only "output" is the ErrorLevel code:

     class SimpleExitCode     {       static int Main(string[] args)       {         int exitcode = 0;         // No argument specified         if (args.Length == 0) return 1;         // Could not parse command line argument         if (!int.TryParse(args[0], out exitcode)) return 2;         return exitcode;       }     } 

Another approach, which works from within a console application or a Windows application, is setting the Environment.ExitCode property:

     // Set the errorlevel to     success System.Environment.ExitCode = 0; 

Exit codes can be negative, but negative values are reserved for system ErrorLevels and should typically be avoided. For example, an ErrorLevel of -1073741510 is returned if the keyboard combination of Ctrl+Break is used during execution of a .NET console application.

Running the generic test

The next step after creating the generic test is to run the test and try it out. The Test View window provides a quick and easy way of running a set of tests. To view the available Team Test windows, from the main Visual Studio menu, select Test image from book Windows. Select Test View.

Select the Ping generic test from the list and click the Run Tests button to execute the test (see Figure 17-5).

image from book
Figure 17-5

As the test is run, the Test Results dialog appears, as shown in Figure 17-6.

image from book
Figure 17-6

You know that the test pings, so the test should pass. If we double-click on the test line that has just passed in the Test Results window, a report of the test run will display. Figure 17-7 shows the output captured from ping.exe. The fields Internal Test Name and Detailed Results File are not applicable to a simple generic test with only a return result and are discussed in the section "Extended Return Results."

image from book
Figure 17-7

Let's see what happens when a test fails. Go back to the Generic Test properties document (Figure 17-4) by double-clicking the Ping.GenericTest file in the Solution Explorer. Change to, save it, and close the editor. Re-run the test from the Test View window (refer to Figure 17-5), and this time the test will fail, as shown in Figure 17-8. Notice that the exit code is 1, which is not 0 and thus indicates a failure.

image from book
Figure 17-8

Quick links are visible on failed test results that conveniently allow you to re-run or debug the original test directly from the Test Results window.

This example has demonstrated how to leverage an existing application to quickly create a test, how to run the test, and how to see the results when it passes or fails. Now you may want to create your own external applications to wrap in a generic test.

Professional Visual Studio 2005 Team System
Professional Visual Studio 2005 Team System (Programmer to Programmer)
ISBN: 0764584367
EAN: 2147483647
Year: N/A
Pages: 220 © 2008-2017.
If you may any questions please contact us: