Creating Tests and Test Plans

In RUP, quality is a way of life. It begins early and continues throughout the complete lifecycle. We all agreed that we wanted to deliver solid software that would give our customers few problems.

We approached testing from two directions for the Elaboration phase. First, Chris and Gary planned to develop and run unit tests. They decided to use the JUnit test framework ^[3] and to try the test-first practice described in eXtreme Programming (see Appendix C). Liz and Jas planned to work on software acceptance tests. They started with the use cases. They also engaged Russell as much as possible to ensure that the tests they developed reflected his expectations.

^[3] See www.junit.org for information on the JUnit test framework.

Test planning proved to be a bit of a sticking point for us. We thought we should have a written plan so we could refer to it as we evaluated the different iterations. We wanted to ensure consistency in our testing across iterations.

Chris volunteered to write the test plan. He started with the Test Plan Template from RUP. The template was large and included many sections that weren't helpful to our project. Chris' first draft had ten pages of words, with little or no content. He ran into one of the problems we often see when people work from templatesthey think all the sections are important and necessary.

Chris and Gary discussed the template and decided on a course of action. They wrote a simple statement describing how the team should approach testing.

Each class will have corresponding unit tests. No code will be checked into version control unless all unit tests pass.

Acceptance tests will be run and will pass before any software is delivered to the customer.

This test plan was small enough to fit on a large Post-it note that we could pin on the wall. It conveyed everything we needed to know. It didn't specify in detail how to write our tests. It didn't specify what tools to use or the structure of the tests. For our four-person project, this was good enough .

Liz and Jas collaborated on defining the initial acceptance tests based on the use cases. Liz then took the lead and produced the acceptance tests. Russell reviewed the tests with the rest of the team and we used the tests to assess the Elaboration iteration plan's success. Liz wrote an acceptance test plan for the Elaboration phase. She was most comfortable writing ideas in a document, but she made sure not to get caught up in making the document perfect. She wrote the test document with enough information so that a tester could run the test and determine if the program behaved as expected. Figure 6.5 is an excerpt from the Elaboration Acceptance Test document. It shows a sequence of tests to run for the basic flow of events of the Open Project Database use case.

Figure 6.5. Excerpt from the Elaboration Acceptance Test document

When the document was created we didn't know the exact commands, message text, or the look and feel of the program. The use-case description provides enough guidance to write the test. The test in Figure 6.5 can be restated as the following sequence of operations through some scenarios of the use case.

1. Open the database AccTest1. You are a member.

2. While AccTest1 is open, choose to open a different database, AccTest2. You are a member.

3. While AccTest2 is open, open AccTest1 again, but try to log in as a user who isn't a member. The application should let you add yourself to the project database.

4. Switch one more time to AccTest2, as a user who isn't a member. This time, don't add yourself, but change to a user who is a member of AccTest2.

5. Finally, re-open AccTest2, but cancel out of the login and exit the program.

This sequence allows a tester to verify a significant number of combinations from the scenario for logging into a PSP project database.

Don't Forget Exploratory Testing

Exploratory testing is a good idea for any project. Regardless of the amount of test planning you do, you are likely to uncover more defects, in a short time, by doing exploratory testing.

Exploratory testing is "purposeful wandering: navigating through a space with a general mission, but without a prescribed route." ^[4] To get started, select a specific subset of the functionality to explore and try a few things to see how well the software meets your expectations. You may find some problems. You may find some surprises . You can scribble some notes and use them to design some more formal test cases and tests.

^[4] From Lessons Learned in Software Testing by Cem Kaner, James Bach, and Bret Pettichord.

Russell and Gary were very good at exploratory testing. With each new build, they were able to quickly uncover major problems in the added functionality.

Unit Testing

Setting up the development environment for unit testing was simple. Chris and Gary downloaded JUnit, installed it, and made the junit.jar file a part of their development project's CLASSPATH. ^[5] It really is that easy!

^[5] Java uses the CLASSPATH environment variable to identify the location of libraries and other files needed to execute a Java program.

Writing the tests and getting them to work is not so easy. Gary had previously used JUnit, but Chris hadn't. Our intention was to write tests for our classes before we wrote the implementation code for the tests. For some types of classes, but not all, this is a straightforward process. We compromised this practice a lot as we proceeded. We wrote tests first for many of our data object classes. For some classes we wrote tests later. Our GUI classes never had unit tests written for them. See Chapter 7 for more details on our unit testing.

It isn't clear to us what effect the compromises we made had on the quality of the final product. There are several practices that have been recently introduced by the agile community. Some of them, like writing tests first, make sense. At the time we are writing this, actual empirical evidence is sparse. There have been a few experiments run on test-first programming (or test-driven design), with mixed results.

Gary recently taught a graduate class in empirical software engineering at Worcester Polytechnic Institute (http://www.wpi.edu). They ran an experiment and found some evidence that test-first was beneficial, but the evidence was not conclusive. ^[6] We think the practice has merit, but suspect there is a significant learning curve before one gets good at it, as is the case with many practices.

^[6] The Web site for the course is www.cs.wpi.edu/~gpollice/cs562-s03.