Developer Testing Issues

XP is heavily reliant on testing. Tests tell us whether we have completed the implementation of some required functionality and whether we have broken anything as a result. On an XP project without extensive design documentation, the tests, along with the code, become a major means of communication. It is not just a case of letting "the code speak to you." The tests must speak to you as well. And they should be listened to over and over again.

Slow Test Execution

Because we rely so heavily on tests to guide development and report on progress, we want to be able to run tests frequently for feedback. We have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.

A purely economic argument is compelling enough by itself. Assuming a developer runs the test suite every ten minutes while developing, they will have run the suite 24 times in a single four-hour programming session. A one-minute increase in test execution time increases development time by 10% (24 minutes)! But the result of slow tests is even more insidious than the economic argument insinuates.

If test execution is too slow, developers are more likely to put off testing, and that delays the feedback. Our preferred model of development is making a series of small changes, each time running the appropriate test suite to assess the impact of the change. If tests run slowly, developers will likely make a number of small changes and then take a "test timeout" to run the tests. The impact of this delayed use of tests is twofold. First, debugging becomes more difficult because if tests fail after a series of changes, identifying the guilty change is difficult. Developers may even forget they made some changes. Murphy's Law says that this forgotten change will most likely be the source of the failing tests.

Second, if tests take so long to run that a developer leaves their desk while a test suite runs, their train of thought will likely be broken. This routine of starting a test suite and then getting up for a stretch and a chat was observed regularly on one project. The lengthy test suite execution time was due to the large number of tests and the relational database access required by each test. This combination became a recurring pattern on a number of projects.

Expensive Test Development

Given the test-first approach taken by XP, the cost of writing and maintaining tests becomes a critical issue. Unit tests can usually be kept fairly simple, but functional tests often require large amounts of fixture setup to do even limited testing. On several projects, we found that the functional tests were taking large amounts of time to write and needed considerable rework every time we added new functionality. The complex tests were hard to understand in part because they contained too much necessary setup. We tried sharing previously set-up objects across tests, but we also found it difficult to write tests that did not have unintended interactions with each other.