3.3 Organizing and Running Tests

Our mini-example has not placed major requirements on the test organization and test execution yet. The two test classes, DictionaryTest and DictionaryParserTest, can easily be restarted over and over again in the same test runner, and the execution time is negligible. However, as our project grows—and with it the number of test classes—this naive approach fails; we need to do some more thinking about test organization and test execution.

Organizing Tests

The first question is, Over how many test classes should I distribute my tests? So far, we packed all tests of an application class into a separate test class. As a rule of thumb, one test class per application class is normally suitable as a starting point, but there are good reasons to deviate from it now and then:

The goal of the test fixture we created in setUp() (see Chapter 2, Section 2.3) is to have an object configuration for all tests of that test class. If certain tests do not need this fixture, then they are extracted to a new test class. In our dictionary example, two groups of tests for the Dictionary class could well form during the further course of development: one needs a Reader as a fixture, while the other doesn't. The result would be two test classes for one application class, for example, DictionaryTest (as before) and DictionaryFromReaderTest.
If the number of tests for a CUT becomes too big, then we should search for groups of tests with common properties and move them into a new test class. This is often a good point for refactoring towards an abstract test superclass, accommodating the common code of the two concrete test classes, as shown in Figure 3.2. A very large number of tests in a test class can also be an indication that the pertaining CUT should be split.

Figure 3.2: A small test class hierarchy.
If a trivial class does not require tests (e.g., DictionaryParserException discussed later in this section), then we would naturally also cut the test class.
Tests referring to a configuration of classes or a subsystem and not to a single class deserve their own test classes.

Another issue worth discussing is whether test classes belong to the same or a separate Java package and whether or not to separate test code from application code. Weighing up the benefits and drawbacks of the conceivable solutions does not lead to a clear judgement:

Test classes in the same package allow access to method with protected or package scope visibility. This can improve the testability, but it also tends to create unstable tests (see also Chapter 8, Section 8.2).

Another benefit is that the tests are near the tested classes. This means that lesser test classes will be "accidentally" overlooked when code is changed or moved. Also, it makes it harder to forget relevant test classes when renaming packages and classes.

In contrast, the separation of application code from test code becomes more difficult, because when separating them one has to rely on a name convention. Test dummy classes can easily be taken as application classes by mistake.
Test classes in separate test packages facilitate the separation of application code from test code. The downside is that non-public classes and methods can be accessed only over auxiliary tools or by the Java reflection mechanism. Whether or not the tests should be accommodated in a parallel hierarchy (e.g., tests.myproj.*) or whether each package should have its personal test package (e.g., myproj.pack1.tests) is simply a matter of taste.
A third solution is to have test code and application code in the same logical package but in different physical paths. In this case, the test package looks like one and the same package from the Java perspective but can easily be separated for deployment purposes. Nowadays most IDEs support this distribution of sources among several base directories.

All three approaches have their specific benefits and drawbacks. The main thing is to use a common standard across the entire team.

Further advantages can be achieved by agreeing on how to address all tests of a package, a subsystem, or the entire project. To this end, AllTests classes represent a quasi-standard. Typically, such a class exists for each package where the test classes are located. This looks as follows in our example:

 import junit.framework.*; public class AllTests {    public static void main(String[] args) {       junit.awtui.TestRunner.run(AllTests.class);    }    public static Test suite() {       TestSuite suite = new TestSuite("Test suite for chapter3.*");       suite.addTestSuite(DictionaryTest.class);       suite.addTestSuite(DictionaryParserTest.class);       return suite;    } }

While the suite() method groups all test classes of the package, the main() implementation offers us a comfortable point to start the test suite without having to worry about parameters. Which one of the three test runners is used for this purpose is again a matter of taste. The important thing is that, when adding, deleting, and renaming test classes, we must not forget to adapt AllTests accordingly.

Depending on the system size and structure, at least one additional AllTests class exists, which groups all other AllTests suites into a test suite, for example:

 package myproj.alltests; import junit.framework.*; class AllTests {    public static void main(String[] args) {       junit.awtui.TestRunner.run(AllTests.class);    }    public static Test suite() {       TestSuite suite = new TestSuite("All tests of MyProject");       suite.addTest(myproj.pack1.AllTests.suite());       suite.addTest(myproj.pack2.AllTests.suite());       suite.addTest(myproj.pack3.sub1.AllTests.suite());       suite.addTest(myproj.pack3.sub2.AllTests.suite());       return suite;    } }

The introduction of additional intermediate hierarchies for AllTests classes can simplify partial testing and updating of larger projects. Figure 3.3 shows a conceivable package and class structure for a fictitious project. Here, myproj.pack3.alltests.AllTests groups all tests from the subpackages of myproj.pack3 together.

Package structure with test classes.

 /myproj/pack1/                                   ClassA.java                                   ClassATest.java                                   ClassB.java                                   ClassBTest.java                                   AllTests.java /myproj/pack2/                                   …                                   AllTests.java /myproj/pack3/subpack1/                                   …                                   AllTests.java /myproj/pack3/subpack2/                                   …                                   AllTests.java /myproj/pack3/alltests/                                   AllTests.java /myproj/alltests/                                   AllTests.java

Figure 3.3: Package structure with test classes.

An alternative approach to the manual maintenance of test suites is to have all test classes of a package or even of an entire application generate and group automatically, by searching all subclasses of junit.framework.TestCase, for example. At first sight, this approach appears easier; the downside is that control over volume, organization, and distribution of all tests is easily lost.^[10] One reviewer put it straight:

Organizing groups of test suites requires some thought and is not a task well-suited to arbitrary automation. The reasons for grouping test suites (other than package-based grouping) is not likely to be something the compiler can figure out. For a very complicated project, suites might be organized to many different things.

I can add nothing to this and prefer the manual maintenance of test suites, myself.

Running Tests

There are numerous opportunities to run a test suite [Gassmann00]:

Before and after refactoring of code.
Before and after adding new functionality.
Before and after integrating modified code into the overall system.
Whenever you want to see a green bar to boost your mood.

In this respect, it is meaningful to select the number of executed tests so that as many test runs as possible are done, as long as this does not give a feeling that your work is hindered by unnecessary waiting. As long as a suite, including all tests of the system, runs in a few seconds, it won't hurt to start the full test set again whenever you make a local change. On the other hand, experience has shown that the runtime for all unit tests will be noticeable sooner or later. This is the reason we try to select a smaller execution granularity, depending on the individual case: subsystem suite, package suite, single test class, and sometimes even a single test case. The following is typical:

Running a single test class during the development of a given class.
Running all tests of a package before and after refactoring actions limited to that package.

At the latest, you should test upon completion of a task and before the integration of modified classes; however, all tests of the system should be run. If you have a feeling that the runtime of the tests hinder you in running the tests regularly, then you could try to reduce the runtime, perhaps by using the techniques described in Chapter 6.

^[10]Java code to automatically find all TestCase subclasses and create a TestAll suite is found in an article by Schneider [00].