Objective 1: Beta Test to Validate That User Expectations Are Met

Objective 1: Beta Test to Validate That User Expectations Are Met

Capturing, Analyzing, and Implementing Change Requests

The first operational version ”the beta release ”of the system was completed and deployed by end of Construction. During Transition you do the beta testing, which provides a lot of user feedback from the beta testers. To ensure that the feedback is useful, actively gather it through interviews, online queries, submitted change requests, or other means. You need to take the time to analyze the feedback, and submit and review change requests so you understand what changes are required before final product release.

Change requests ( mainly defects and beta test feedback) are the major planning input for continuing development. Mostly, the change requests are only for minor system tweaking, such as fixing minor bugs , enhancing documentation or training material, or tuning the performance. Sometimes additional features must be added, requiring that you work with requirements, analysis and design, implementation, and testing (see the section Iteratively Develop a Complete Product That Is Ready to Transition to Its User Community, in Chapter 8).

It should be noted that adding features this late in the project could indicate failure in earlier phases, but could be well-motivated especially for very large and complex systems. In most cases you should refrain from adding new features and instead postpone them to the next development (evolution) cycle. In some cases, however, the system may not deploy successfully without the additional feature.

As you implement change requests, it is essential to have in place a good Configuration Management process as well as comprehensive regression testing. At this stage, builds with incorrect file versions or missing files are common sources of defects. Good CM practices and tools radically reduce these types of errors, and comprehensive regression testing rapidly identifies any introduced defects.

During Transition, you need to spend a fair amount of time improving documentation, online Help, training material, user's guides, operational guides, and other supporting documentation. It is important that actual beta testers properly test these elements in the target environment.

Transition Testing

The focus of testing during Transition shifts toward improving quality and avoiding regression. Additionally, there is often a requirement for formal acceptance testing, which may involve a repeat of all or part of the system-level tests. In planning for Transition testing, you should provide effort and resources for the following:

  • Continued test design and implementation to support ongoing development.

  • Regression testing, which will require variable effort and resources, depending on the chosen approach; for example, retest everything or retest to an operational profile.

  • Acceptance testing, which may not require the development of new tests.

As defects are fixed and beta feedback is incorporated, successive builds are tested using a standard test cycle:

  • Validate build stability. Execute a subset of tests to validate that the build is stable enough to start detailed test and evaluation.

  • Test and evaluate. Implement, execute, and evaluate tests.

  • Achieve your test objectives, or, in RUP parlance, achieve an acceptable mission. Evaluate test results against testing objectives and perform additional testing as necessary.

  • Improve test assets. Improve test artifacts as needed to support the next cycle of testing.

When the system is deemed fit to undergo acceptance testing, a separate testing cycle is performed focusing on executing tests and evaluating results.

Patch Releases and Additional Beta Releases

If serious defects that prevent effective beta testing are found, you may need to produce a patch release (special bug-fix release installed on top of the current baselined release) and make it available to beta customers for download. Often these patch releases may be available only to specific beta testers to address their special areas of concern or to block defects they are encountering (although sometimes they go to all beta testers).

As described in an earlier section, Transition and Iterations, there may be more than one iteration in Transition. If so, you typically produce a new release that is distributed to all beta testers, such as Beta 1, Beta 2, and so on. You need to ensure that enough beta testers are upgrading to the new release to provide sufficient feedback.

Metrics for Understanding When Transition Will Be Complete

It is not always obvious when you can end Transition. To help determine when Transition will be complete, you can analyze defect metrics and test metrics, among other things. This analysis can help you answer questions such as, "When will the quality be good enough?", "How many more defects can we expect to find?", and "When will we have tested all functionality?"

Let's look at a small sample set of metrics you may find useful: defect metrics and test metrics.

Defect Metrics

For defects, you should track

  • How many new defects are found each day.

  • How many defects are fixed each day.

The important thing is to focus on the trend rather than the actual number of defects. Figure 9.4 shows that the number of open defects increased until February 10, and as you moved more and more developers to fix defects, the code improved in quality. You closed more defects than you opened, leading to a rapid decrease in the number of open defects. Through trend analysis of this figure, you can estimate that 20 open defects will be reached around March 9, and another week would be needed to come down to below 5 defects. However, it is common that as you approach 0 open defects, the defect-fixing rate will slow down ”this needs to be factored into your estimates.

Figure 9.4. Trend Analysis of Defects. By analyzing a defect trend, you can predict when a certain threshold value of open defects will be reached. If you plan to release the product once there are less than 20 defects, the graph shows that this will likely occur around March 9.

graphics/09fig04.gif

Test Metrics

Another important set of metrics of great value is test metrics, such as test completion. If, for example, only 60 percent of planned tests are completed, you should expect to find many more defects once testing is completed. You can predict how many new defects can be expected by determining how many defects are typically found per test case and multiplying that by the number of test cases yet to be executed and analyzed . This can be expressed with the formula:

D Remaining = D Average x ( TC Total - TC Executed )

Where

D Remaining = number of defects yet to be found

D Average = average number of defects found per test case

TC Total = total number of test cases

TC Executed = number of test cases executed so far

You do not want to deliver a system that has not been properly tested. By analyzing the trend of completed and successful test cases, you can predict when you will have completed the test effort, yet another data point allowing you to decide when it is appropriate to end Transition.



The Rational Unified Process Made Easy(c) A Practitioner's Guide to Rational Unified Process
Programming Microsoft Visual C++
ISBN: N/A
EAN: 2147483647
Year: 2005
Pages: 173

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net