Reporting and Release Processes

This section discusses the last two phases in the overall testing process: reporting and release to production.

Reporting the Results

After the test team completes a successful execution of the required test cases, a test report is prepared to list the remaining open bugs. This final test report is called the Release to Production report. This report describes the areas that were tested and summarizes the testing results. It also lists bugs or issues that were postponed to future versions. The report also highlights the impact on business if the remaining bugs are not fixed, and includes a recommendation on whether to proceed with production. The completed report is presented to the project management teamto help that team evaluate whether the product can be released to production or whether the product must be fixed.

Release to Production

After the project management team receives the Release to Production report, the teams involved in the migration project (development team, test team, and program management team) meet to discuss whether to release the product to production.

If the teams agree that the product meets all the requirements of the project, it canbe released to production. If not, the outstanding issues are reviewed and fixed as part of the standard triage and fix process described earlier.

The following sections discuss the different criteria that the teams should take into account when deciding whether the implementation is ready for release.

Test Pass/Fail Criteria

A test case passes if the actual result matches the expected result. If the actual result does not match the expected result, it is treated as a failed test case.

If a test case fails, it is not assumed that the feature is defective. For example, misinterpretation of project documentation, incomplete documentation, or inaccurate documentation can cause failures. Each failure is analyzed to discover its cause, based on actual results and the results described in project documentation.

Pass criteria are as follows :

• All processes run with no unexpected errors.

• All processes finish in an acceptable amount of time, based on benchmarks defined in the functional specification.

• Load tests show that a satisfactory level of capacity is in place and that appropriate steps can be taken to scale out the system when necessary.

Test Suspension Criteria for Failed BAT and Test Execution

The test team may suspend partial, or full, testing activities on a given build if any of the following events occurs:

• A systems engineer is unable to install the new build or a component.

• A systems engineer is unable to configure the build or a component.

• A major feature has a fault that prevents a significant area from being tested.

• The test environment is not stable enough to trust the test results.

• The test environment is very different from the expected production environment, so test results cannot be trusted.

Resumption Requirements

The test team may resume testing if:

• The problem that suspended the testing is corrected.

• The development and test teams agree that fixing the bug can be postponed until the next iteration.

Release Criteria

The release criteria depend on the severity of bugs that are open. As noted earlier, these criteria are measured on a scale of 1 to 4, where level 1 represents the highest severity, and level 4 represents the lowest severity.

The following are some possible criteria for allowing the code to be released to production:

• There are no open bugs with a severity of 1 or 2.

• Test cases scheduled for both integration and system test phases have successfully passed ( especially test cases that cover major functionality or that were givena high priority or a medium priority).

• The final regression test has successfully passed.

Note  

The criteria for release vary widely, depending on both the architecture and the natureof the migrated application.