Testing with Iterative Lifecycle Models

Testing with iterative lifecycle models, such as the Rational Unified Process, occurs during, and especially near the end of, each iteration. As explained in Chapter 2, each iteration produces a specific, tested, executable release that addresses specific risks and requirements. These releases are then baselined in a configuration management tool. As testing progresses, defects are entered into a defect-tracking tool. The defects are prioritized and, in most cases, corrected and retested by the end of that iteration. Some lower-priority defects can be deferred to subsequent iterations. Figure 11-2 illustrates this process. Coding begins in iteration 1. Testing begins during the iteration and concludes by the end of the iteration. Defect correction begins near the end of the iteration. The process continues for each subsequent iteration.

Figure 11-2. Testing and defect correction progress during iterations

Advantages of Testing with Iterative Development

Numerous improvements to the testing process are made possible through the use of iterative development. The following list explains each of these advantages:

• Testing begins much earlier in the project lifecycle compared with traditional Waterfall lifecycle methods. Testing could even begin with the first executable baselined iteration in the Inception phase. In most cases, testing is more likely to begin in the Elaboration phase, since Inception phase iterations are the only ones that may not have an executable release. Testing so early in the lifecycle creates the following benefits:

  • Testing can proceed at a deliberate, sustainable pace. In the first few iterations, the amount of testing is quite light. Each iteration implements a subset of new requirements. Thus, testing for each iteration is composed of functional testing of all the new requirements implemented in that iteration, plus regression testing on requirements implemented in prior iterations. Because early iterations require little regression testing, testing can easily be accomplished in the iteration's time frame. As the project's lifecycle progresses, testing requires more resources as the amount of functionality that must be regression-tested increases.

  • As testing begins and slowly ramps up, the testing personnel gain experience with the tools and testing process. They can then apply this knowledge to the testing of subsequent iterations. They can fine-tune the process and use of the tools to be more effective as a result.

  • Early product deployment is easier, because testing begins so early in the project lifecycle. If a decision is made during the project to deploy early, the majority of the functionality developed at any one point has already been tested. The result is a higher-quality product. Or, at the very least, the problems are known, and users can be forewarned.

• As explained in Chapter 2, iterative development lets you develop "riskier" requirements first. Therefore, requirements that represent significant technical risks are implemented and tested in the earlier iterations. Accordingly, this means that requirements that are particularly difficult and complex to implement are afforded a longer period of time to correct the defects discovered.

• Because the testing process is repeated for each iteration, the testing team can give feedback to the development team. The development team can then incorporate the feedback into the process. For example, if the development team is using constructs in implemented code that are incompatible with automated testing tools, this is discovered early, and adjustments are made. If certain types of errors are discovered frequently, the cause can be determined and the process adjusted to prevent them from reoccurring.

Prerequisites for Testing with Iterative Lifecycle Models

Before implementing a testing process on projects that use iterative lifecycle models such as the RUP, the practitioner should be aware of the testing team's needs. These needs do not differ significantly from testing teams on Waterfall lifecycle projects, but some needs take on an increased level of urgency.

Requirements Management

For each iteration, only a subset of the complete set of functional requirements are implemented and tested. Furthermore, at the conclusion of each iteration, while planning the next iteration, the requirements assigned to subsequent iterations may change. Finally, requirements themselves can change. In Waterfall-based processes, from a testing perspective, this is not a problem, because all functional requirements are considered ready for testing all at onceafter they are all implemented and integrated to create the complete system. For an iterative lifecycle model, this requires careful tracking of requirements. Because testing begins earlier in the project lifecycle, requirements are more likely to change. Also, it's important to know for each given iteration exactly which requirements have been implemented and which ones have not.

This means that iterative lifecycle models require some sort of automated requirements management tool. Testing personnel need to quickly determine which requirements are implemented for each iteration. They need to know this information as soon as it is available so that test plans and scripts can be finalized. They need to be informed quickly if any requirements change so that they can review any test artifacts to determine how the change affects the artifacts and make any adjustments to the test artifacts if needed.

I also like to use the requirements management tool to track, for each functional requirement, which requirements have been implemented and which have passed testing. As the iterations progress, I often find it helpful to post graphs and charts of the progress in some common area where project members can see them. They can even be used as input to calculate Earned Value if your project uses it.

Configuration Management

It's also vital to have robust configuration management tools and processes in place for testing with iterative lifecycles. This includes workspace management. Because testing and development occur simultaneously, the configuration management tools and processes must do the following easily and intuitively:

• A complete, identifiable baseline of the just-completed build, including all the executables produced, needs to be created and maintained. This provides a stable release for the testing personnel to work with.

• Ongoing new development for the subsequent iteration must be isolated from the previous baseline. Developers need freedom to perform their work without fear of disrupting the test process. The test team must have a stable baseline that is completely frozen from any changes.

• The test team needs to determine exactly which requirements were delivered in the baselined build, as well as which defects (if any) were supposedly fixed in that baselined build.

• Depending on the organization's process, it may be important for developers to make emergency bug fixes to the baselined build without actually changing the baseline itself or mixing in newly developed functionality from the subsequent build. In most configuration management tools, this is called a branching capability. Some organizations choose to simply complete as many tests as possible and fix them late in the iteration.

Change Management

A defect tracking system that is easily accessed by testing personnel and developers must be in place. It should track all information related to the defect, including how it is reproduced. In particular, it should identify the baselined build in which the defect was discovered, along with an estimate, or rating, of the defect's severity.

It's important for the testing personnel to thoroughly document the exact steps needed to reproduce the problem. Screenshots of any error messages or screen settings are also useful. When the testers enter a defect into a defect tracking system, they should word the defect's headline or title carefully. The tester should use the headline or title to capture the essence of the problem and also get the team's attention. You don't want the developers to be unable to reproduce the defect because the description is too vague. This wastes the time of both testing personnel and developers.

Staffing

Testing personnel are needed much earlier on projects that use iterative lifecycle models. However, unlike the "big bang" of the Integration and Test phase of the Waterfall lifecycle, staffing can ramp up more slowly, starting with senior test personnel for planning the testing process. As more iterations are completed, more test personnel can be added as the amount of code to be tested increases. This may be alleviated somewhat through the use of automated testing tools, which are discussed a little later in this chapter.

Because testing begins much earlier in an iterative lifecycle, testing personnel and developers work together closely for much of the project. It is vital that these two groups work well together. It's not uncommon for testing personnel to need assistance from the developers while they are testing an iteration. This informal communication is good and should be encouraged. However, the testing personnel need to remember that the developers are on fairly tight deadlines as they work to produce the next iteration.

Feedback for the Iteration Planning Process

As the testing results become available for a given iteration, the project manager should examine the defect reports that are available from the testing. As part of planning the iterations, the project manager should allow some time to correct bugs as well as test and implement another subset of the requirements. It may be unrealistic to expect to have enough resources to fix every bug. In that case, the severity rating on each defect report can be used to prioritize which ones will receive attention in the next iteration. Maintaining a "queue" of the remainder of the defects can be useful if a developer finishes his work for an iteration early, or an extra person becomes available. This keeps everyone busy and uses staff more efficiently. It also reduces or eliminates an extensive "cleanup" or defect correction activity later in the project lifecycle. Any defects that did not receive attention due to time constraints should be documented in the product's release notes. Although defects are undesirable, the fact that they are documented tells the customer that you've already tested the product and are at least aware of them.

To illustrate this process, consider Figure 11-2, shown earlier. During iteration 1, a build becomes available containing functionality that can be tested. The testing begins, and defect reports are submitted into a tracking tool. Toward the end of iteration 1, defects are prioritized, and correction and retesting begin. Any defects not corrected are examined, prioritized, and scheduled for correction in the next iteration. In the next and subsequent iterations, the process repeats until the end of the project.