Wiring to Existing Test Systems

Supporting existing test frameworks by linking a generic test to the other test framework is one of the scenarios mentioned in the section "Creating an External Testing Tool," above. If you already have a test framework in place and you want to use it with Team Test, you need a tool for converting the result output from the existing test framework to a format the generic test engine recognizes.

As time progresses, more conversion tools will be made available online. Until one is available for the test framework you're currently using, you may need to manually create one. Thankfully, most test frameworks can produce XML output that can be transformed into the generic test results schema.

The example provided in this section combines the "Script Host Example" and "Extended Return Results" to wire together an NUnit 2.0 test suite to a generic test. Because NUnit produces an XML results file, and the generic test engine accepts an XML file as results, all you need is an XML transformation from one schema to another. This is easily implemented with some JScript and an XSL file.

This is an example and is not intended to be an all-encompassing transformation solution. You may encounter more complex results that cannot be easily transformed by an XSL. In this case, you'd need an interpreter application that reads one format, interprets the results, and produces the resulting XML file. The tool could also create elaborate reports that could be added to the <DetailedResultsFile> tag and shown in the summary report, shown in Figure 17-10.

Creating a wire to NUnit

To see a generic test run an NUnit test suite and then convert the NUnit results to a generic test XML result, just follow the steps outlined in the following sections. Each step reflects material drawn from various earlier parts of the chapter.

Flow of operation

This example is designed to use all the extensible points in the generic test properties document (see Figure 17-10).

Figure 17-10

Here is what happens in this example:

1. Team Test starts the generic test.

2. Team Test sets up the process's environment by setting the environment variables and copying the deployment files.

3. The generic test starts the Windows Scripting Host, which executes the JScript file.

4. The script runs NUnit with the appropriate command-line options.

5. NUnit performs its tests and outputs an XML file with its test run results.

6. The script uses MSXML to pull in the NUnit XML file and the XSL transform file and perform the transformation.

7. An XML file conforming to the generic test results scheme is written to disk as Generic Test XML Results.xml.

8. The script ends.

9. The generic test reads the results from Generic Test XML Results.xml file.

10. The generic test type returns the results back to the test framework.

11. The test run is complete.

Setup and run

Here's how you would carry out this workflow:

1. Install NUnit 2.0. NUnit is a free download and will install the necessary C# sample files.

2. Run the included NUnit C# Example, designed to be used with Visual Studio .NET 2003. If you want to use Visual Studio 2005 (.NET 2.0), you need to apply a workaround to replace it with a different runtime version (posted on http://www.blogs.msdn.com/jamesnewkirk/archive/2004/07/05/173513.aspx). Load the solution C:\Program Files\NUnit 2.2\src\ nunit.sln and follow the directions to reset the references and build at least the C# project (the other projects can be removed from the solution).

3. Copy the following below to a deployment directory. For this example, use C:\Temp\NUnitExample.

4. Create a new generic test. Follow the steps under "Creating a generic test" earlier in this chapter.

5. Change the properties as outlined in the following table. Note: The actual directory NUnit installs in is different depending on what version of NUnit you have installed.

   Field/Property	Value
   Specify an existing…	C:\WINDOWS\system32\cscript.exe
   Command line arguments…	/nologo RunNUnit.js "C:\Program Files\NUnit 2.2\samples\csharp\bin\Debug\csharp-sample.dll"
   Additional files…	RunNUnit.js
   Environment Variables
   NUnitExe	C:\Program Files\NUnit 2.2\bin\ nunit-console.exe
   XslTransformFile	C:\Temp\NUnit to Team Test.xsl
   Working Directory	%TestOutputDirectory%
   Redirect standard output…	Checked
   Display execution application…	Unchecked
   Exit test…	30000
   Summary Results File (checkbox)	Checked
   Summary Results File (field)	%TestOutputDirectory%\Team Test NUnit Results.xml

6. Run the NUnit Generic Test. For details, see the section "Running the generic test."

7. View the Results Report (see "Example Summary Report XML File" and Figure 17-10). Notice that in this case, there are several Inner Tests because each NUnit "test-case" is translated to an Inner Test.

JScript conversion code

We used JScript in this example because many test frameworks are tied together via scripts like this one. The script deals mostly with files and XML, for which the Windows Script Host is optimized. It can be quickly and easily modified, and deployment is easy with just the one .js file.

This simple DOS batch file will sufficiently set the needed environment variables and test the script on the command line, separate from the generic test framework. It is recommended that you get this combination working first from the Windows command line (cmd.exe), and then use the script directly from the generic test (without the batch file).

     @echo off     set TestOutputDirectory=C:\Temp\TestOutputDir     set NUnitExe=C:\Program Files\NUnit 2.2\bin\nunit-console.exe     set XslTransformFile=C:\Temp\NUnit to Team Test.xsl     cscript.exe /nologo RunNUnit.js "C:\Program Files\NUnit     2.2\samples\csharp\bin\Debug\csharp-sample.dll"

The script performs the following major operations:

• Checks the environment variables and command-line options

• Executes NUnit

• Transforms the XML output from NUnit to a Generic Test Results XML file conforming to SummaryResults.xsd scheme

• Writes the result to an output XML file that the generic test framework will pick up to display a complete set of test results

If an error occurs along the way, the app will exit and set the ErrorLevel code. Because it was specified above that a Summary Results XML file would be used, the exit code is just ignored by the test framework. However, it is good practice to set the exit code. This sample could be considerably reduced if the script just ran NUnit, looked at the top-level suite pass/fail result from the resulting XML file, and set the exit code without transforming the XML output for extended result information. This is a good task to try out to become more familiar with the workings of the generic test type.

     // Runs an NUnit test suite and converts to Team Test     Main();     function Main()     {       // Obtain access to the Windows Shell       var WshShell = WScript.CreateObject("WScript.Shell");       // Collect the environment variables available to this process       var WshSysEnv = WshShell.Environment("PROCESS");       // Retrieve the needed parameters from environment variables       var TestOutDir = WshSysEnv("TestOutputDirectory");       var NUnitExe = WshSysEnv("NUnitExe");       var XslFile = WshSysEnv("XslTransformFile");       // Verify the environment variables are available       if (TestOutDir == "")       {WScript.Echo("Error: Could not find %TestOutputDirectory"); WScript.Quit(3);}       if (NUnitExe == "")       {WScript.Echo("Error: Could not find %NUnitExe"); WScript.Quit(3);}       if (XslFile == "")       {WScript.Echo("Error: Could not find %XslTransformFile"); WScript.Quit(3);}       // Build the file paths       var NUnitOut = TestOutDir + "\\NUnit Test Results.xml";       var TeamTestOut = TestOutDir + "\\Team Test NUnit Results.xml"       // Make sure the command line arguments are specified       if (WScript.Arguments.length == 0)       {WScript.Echo("Syntax: RunNUnit.js TargetDll [TargetDll..]"); WScript.Quit(1);}       // Build the list of .NET assemblies to be tested       var args = "";;       for (var i = 0; i < WScript.Arguments.length; i++)         args += " \"" + WScript.Arguments.item(i) + "\"";       // Execute NUnit       var cmdline = NUnitExe + args + " /xml=\"" + NUnitOut + "\"";       WScript.Echo("Executing NUnit Test Suite&");       WScript.Echo(cmdline);       var exe = WshShell.Exec(cmdline);       // Wait for NUnit to finish       while (exe.Status == 0) WScript.Sleep(100);       // If there was an error, abort       if (exe.ExitCode != 0)       {WScript.Echo("Error: NUnit execution error: " + exe.ExitCode); WScript.Quit(5);}       // Perform the XML transformation       WScript.Echo("Transforming NUnit results to Team Test results.");       var TransformedResults = PerformTransform(NUnitOut, XslFile);       if (TransformedResults == "") WScript.Quit(2);       // Save the transformation output       WScript.Echo("Writing Team Test XML output.");       WriteTextFile(TeamTestOut, TransformedResults);     }     // Applies an XSL transform to an XML document     function PerformTransform(xmlfile, xslfile)     {       // Create the XML data source       var xml = new ActiveXObject("Msxml2.DOMDocument.3.0");       // Load the XML source data       xml.async = false;       xml.resolveExternals = false;       xml.load(xmlfile);       // Verify the XML source data is valid       if (ParseOkay(xml))       {         // Create the XSL utility objects         var xslt = new ActiveXObject("Msxml2.XSLTemplate.4.0")         var xslDoc = new ActiveXObject("Msxml2.FreeThreadedDOMDocument.4.0");         var xslProc;         // Load the XSL source data         xslDoc.async = false;         xslDoc.resolveExternals = false;         xslDoc.load(xslfile);         // Verify the XSL source data is valid         if (ParseOkay(xslDoc))         {           // Perform the XSL transformation           xslt.stylesheet = xslDoc;           xslProc = xslt.createProcessor();           xslProc.input = xml;           xslProc.transform();             // Plase the XSL transform output             return xslProc.output;           }         }         return "";      }      // Write text to a file      function WriteTextFile(filename, contents)      {         var fso = new ActiveXObject("Scripting.FileSystemObject")         var f = fso.OpenTextFile(filename, 2, true)         f.Write(contents);         f.Close();       }       // Verify that the XML is valid, if not show a message       function ParseOkay(xmldoc)       {         if (xmldoc.parseError.errorCode != 0)       {           var err = xmldoc.parseError;           WScript.Echo("Parsing Error:\n" + err.url + "\n" + err.reason + "On line " +             err.line + " at " + err.linepos + "\n" + err.srcText);           return false;       }       else return true;     } 

See the earlier section "Scripting host example" for a good MSDN Library URL with reference information on all the scripting objects used in the code.

The heart of the operation is the XSL transformation. This is a basic transformation designed for the test suite sample code that ships with NUnit 2.0. Note that NUnit reports a test's "success" as true or false (see the "TestResult" template element), which is easily translated into a Passed or Failed result:

     <?xml version="1.0" encoding="utf-8" standalone="no"?>     <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">       <xsl:template match="/test-results">         <SummaryResult>           <TestName>NUnit test on: <xsl:value-of select="@name"/></TestName>           <xsl:for-each select="test-suite[1]">             <TestResult><xsl:call-template name="TestResult" /></TestResult>         </xsl:for-each>         <ErrorMessage></ErrorMessage>         <DetailedResultsFile></DetailedResultsFile>         <InnerTests>           <xsl:for-each select=".//test-case">             <InnerTest>               <TestName><xsl:value-of select="@name"/></TestName>               <TestResult>                     <xsl:if test="@executed='True'">                       <xsl:call-template name="TestResult" /></xsl:if>                    <xsl:if test='@executed="False'">NotExecuted</xsl:if>                  </TestResult>                  <ErrorMessage>Message: <xsl:value-of select="failure/message"/>      Stack Trace: <xsl:value-of select="failure/message"/>                 </ErrorMessage>               </InnerTest>             </xsl:for-each>           </InnerTests>         </SummaryResult>       </xsl:template>       <xsl:template name="TestResult">         <xsl:if test="@success='True'">Passed</xsl:if>         <xsl:if test="@success='False'">Failed</xsl:if>       </xsl:template>     </xsl:stylesheet></para>

Combine all these elements and you have a conversion utility that will run an existing test framework and incorporate the results into Team Test. With simple modifications, most existing test frameworks can be run using this same technique.