Extended Return Results | Professional Visual Studio 2005 Team System (Programmer to Programmer)

A basic external application that is run by a generic test communicates back to the Team System test engine by means of a single exit code indicating a passed or failed result. A more advanced alternative is for your application to create an XML file as part of its output, with detailed results of its run that conforms to the SummaryResult.xsd scheme. If you use an XML results file, you can pass back more information regarding the state of the external application's test run.

An XML results file adds the following additional fields:

Return types—values include Passed, Completed, PassedButRunAborted, Pending, InProgress, Warning, Disconnected, NotExecuted, NotRunnable, Failed, Inconclusive, Error, and Aborted
Path to an additional log file
Verbose test name
Custom error message
Inner Tests, each with its result

To specify that your generic test will return an XML results file that conforms to the SummaryResult.xsd scheme (more about this below), enter the name of the file in the Results Settings section at the bottom of the Generic Tests properties page and check the Summary Results File checkbox. A typical entry here would look like %TestOutputDirectory%\GenericTestOutput.xml, where %TestOutputDirectory will be replaced with the appropriate directory used in the running of the external application. Be sure to store your output file using this environment variable's path in your application.

Summary report XML schema

The schema for the results file is shown in the following code. This schema file, part of the standard files installed with Visual Studio Team System, is called SummaryResult.xsd and is placed in the Microsoft Visual Studio 8\Xml\Schemas directory.

     <?xml version="1.0" encoding="UTF-8"?>     <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"      xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" elementFormDefault="qualified"      attributeFormDefault="unqualified"       <xs:element name="SummaryResult">         <xs:annotation>           <xs:documentation>Describes summary result for tests</xs:documentation>         </xs:annotation>         <xs:complexType>           <xs:sequence>             <xs:element name="TestName" role="xs:string" minOccurs="1" maxOccurs="1" /> />             <xs:element name="TestResult" role="testResultType" MinOccurs="1"              maxOccurs="1" />             <xs:element name="ErrorMessage" role=&"xs:string" minOccurs="0"              maxOccurs="1" />             <xs:element name="DetailedResultsFile" role="xs:string" minOccurs="0"               maxOccurs="1" />             <xs:element name="InnerTests" minOccurs="0" maxOccurs="1">>               <xs:complexType>                 <xs:sequence>                   <xs:element name="InnerTest" minOccurs="0" maxOccurs="unbounded">                     <xs:complexType>                       <xs:sequence>                         <xs:element name="TestName" role="xs:string" minOccurs="1"                          maxOccurs="1"/>                         <xs:element name="TestResult" role="testResultType"                          minOccurs="1" maxOccurs="1" />                         <xs:element name="ErrorMessage" role="xs:string" minOccurs="0"                          maxOccurs="1" />                         <xs:element name="DetailedResultsFile" role="xs:string"                           minOccurs="0" maxOccurs="1" />                        </xs:sequence>                      </xs:complexType>                     </xs:element>                   </xs:sequence>                 </xs:complexType>               </xs:element>             </xs:sequence>           </xs:complexType>          <xs:key name="InnerTestNameMustBeUnique" msdata:PrimaryKey="true"></para>              <xs:selector xpath=".//InnerTest" />              <xs:field xpath="TestName" />            </xs:key>          </xs:element>          <xs:simpleType name="testResultType">          <xs:restriction base="xs:string">          <xs:enumeration value="Aborted"/>          <xs:enumeration value="Error"/>          <xs:enumeration value="Inconclusive"/>          <xs:enumeration value="Failed"/>          <xs:enumeration value="NotRunnable"/>          <xs:enumeration value="NotExecuted"/>          <xs:enumeration value="Disconnected"/>          <xs:enumeration value="Warning"/>          <xs:enumeration value="InProgress"/>          <xs:enumeration value="Pending"/>          <xs:enumeration value="PassedButRunAborted"/>          <xs:enumeration value="Completed"/>          <xs:enumeration value="Passed"/>         </xs:restriction>       </xs:simpleType>     </xs:schema>

In a single summary result file, there must be a TestName and TestResult for the master test and for each (optional) InnerTest that exists. All other fields/tags are optional. The simple type testResultType defines all the possible results of a test, which obviously provides much more granularity than the two states, Passed and Failed, that are used with exit codes.

Example summary report XML file

Here is an example summary report XML file. Notice that the test can be given a friendly verbose name:

     <SummaryResult>       <TestName>Manual Summary Results Example</TestName>       <TestResult>Inconclusive</TestResult>       <ErrorMessage>This is the error that caused the test to fail.</ErrorMessage>       <DetailedResultsFile>C:\Temp\TestOutput\TestOverview.html</DetailedResultsFile>       <InnerTests>         <InnerTest>           <TestName>First Inner Test</TestName>           <TestResult>PassedButRunAborted</TestResult>           <ErrorMessage>A bogus test message.</ErrorMessage>           <DetailedResultsFile>C:\Temp\TestOutput\TestPass01.html</DetailedResultsFile>         </InnerTest>         <InnerTest>           <TestName>Second Inner Test</TestName>           <TestResult>Passed</TestResult>           <ErrorMessage>A bogus test message.</ErrorMessage>           <DetailedResultsFile>C:\Temp\TestOutput\TestPass02.html</DetailedResultsFile>         </InnerTest>       </InnerTests>     </SummaryResult>

This XML file produces the summary results shown in Figure 17-9.

image from book
Figure 17-9

When an inner test is selected with a detailed results file (as defined in the <DetailedResultsFile> tag) the contents will be displayed in the embedded Internet Explorer under "Summary File." Because the file is displayed in an IE control, it can be a full HTML file or XML with XSL for rich display results (including images, like screen shots). A common way of producing a summary file is to perform an XSL transform over your results to create a nicely formatted HTML report. An example of this is included in the book's source code.

Important

The detailed results file for the entire test (the line containing TestOverview.html) is shown only as a property of the generic test and is not rendered inside the test result's IE control. So it is recommended that this property is not used and you use the DetailedResultsFile tag inside the Inner Test instead.