Automating Tests

The value of an application's test suite is dependent on how easy it is to run. There will be many individual test classes, and it must be possible to run them all in a single operation. While each test run should report success (meaning success of all tests) or failure (meaning failure of one or more test), it should also create a log file detailing any failures, to provide a basis for analysis and debugging.

Fortunately, if we implement all application tests as JUnit test cases, there's an easy way to achieve this level of test automation and to integrate test runs into the application build process. The Ant build tool integrates smoothly with JUnit. It's possible to use Ant to run a set of tests, log the results and provide a simple report indicating whether all tests were passed. The following Ant task, for example, looks under an entire source tree and runs as a test any class with a name matching XXXXTestSuite.java. Success or failure will be displayed to the Ant console, while a detailed log will be created.

    <target name="fwtests" depends="build-fwtests">      <tstamp/>      <mkdir dir="${junit.reports.dir}" />      <mkdir dir="${junit.reports.dir}/framework-${DSTAMP}-${TSTAMP}" />      <junit printsummary="yes" haltonfailure="yes">        <classpath ref />        <formatter type="plain" />        <!-- Convention is that our JUnit test classes have names such as             XXXXTestSuite.java        -->

        <batchtest             fork="yes"             todir="${junit.reports.dir}/framework-${DSTAMP}-        ${TSTAMP}">        <fileset dir="${framework-test.dir}">            <include name="***/*TestSuite.java" />        </fileset>      </batchtest>

    </junit>    </target> 

Please refer to the Ant documentation for more details, but the most important Ant task in this example is the <batchtest> task, which runs a number of JUnit test cases. In this example, the <fileset> subelement specifies a wildcard (**/*TestSuite.java) that selects the names of the test classes we wish to run. The todir attribute of the <batchtest> task specifies a directory to which a log should be written.

Using Ant targets such as this to run a test means that we can easily execute all the tests for a set of packages or an entire application with a single command. We can run all the tests after each change or bug fix, or daily.

To ensure that tests are run regularly, it's essential that tests should execute quickly. As a rule of thumb, all tests should be executed within ten minutes: if possible, much quicker. If tests can't be executed within this time scale it may be possible to refactor the test suite to improve its performance. If some tests will inevitably take longer, consider splitting the test suite into tests that provide immediate feedback and tests that can be run overnight.