Expanding Coverage of the Executable Tests

This problem doesn't make the executable test implemented by direct calls bad, it just means these tests can't find a significant population of potential defects. You absolutely want to get the executable tests running through direct calls first. These will both find defects and demonstrate the progress of the project toward the customer's objectives, which is what acceptance testing is all about.

Whether you then spend the effort to get the tests to run through other interfaces depends on the team's assessment of risk for not doing so. The system may be designed so very little code needs to be duplicated in the application test class (see the description of an XP team at BoldTech Systems at the end of this chapter), in which case the risk may be quite small.

The good news is that when you do need to run the tests through some other mechanism than direct calls, you don't need to make any changes whatsoever to the executable tests, since they're completely independent of the interface with the system being tested.

You may have a choice among several methods to interface the tests to your system, other than direct calls. Usually, the farther out toward the user interface you get, the more difficult it becomes.

Consider, for instance, a Web-based system like XTrack. Leaving the user's browser program aside, the system may still consist of discrete programs, such as the Web server itself, an application server, servlets, CGI scripts, database servers, backend hosts, and so on. The easiest way to interface your tests (after direct calls) may be to bypass, for instance, the Web server, and drive the servlet, CGI script, database server, and so on, directly. While this allows you to test all the code in, say, your servlets, you're still bypassing code in the Web server and application server, not to mention the communications layer (Internet) and the browser.

If you move back in the chain, so you're driving the tests through the Web server, you can cover more of the code with tests, but with greater effort because interfacing to the Web server using HTTP protocol will probably be harder than just calling servlets. Going back another step and driving the test through the browser covers even more of the code but is even more difficult, because you have to deal with the GUI thing: mouse clicks, keystrokes, and screen layouts.