Section 8.2. Test Execution

8.2. Test Execution

The software testers begin executing the test plan after the programmers deliver a build that they feel is feature complete. This is referred to as the alpha build . The alpha should be of high qualitythe programmers should feel that it is ready for release, and as good as they can get it. This build should have been code reviewed (see Chapter 5) and should have passed unit tests (see Chapter 7); it should have already been minimally functionally tested by the development team, as well.

There are typically several iterations of test execution . The first iteration focuses on new functionality that has been added since the last round of testing. (If this is the first time this software product has been tested, then every test case is executed.) If no defects are uncovered that are considered high enough priority to fix (see below about defect triage), then the testers move on to perform a regression test .

A regression test is a test designed to make sure that a change to one area of the software has not caused any other part of the software that had previously passed its tests to stop working. Regression testing usually involves executing all test cases that have previously been executed. In other words, it's not enough to verify that the software has been altered: it's also necessary to ensure that the change did not break any other part of the software that previously worked.

It is rare for no defects to be uncovered in the first test iteration. Usually, some test cases fail and defects must be reported. Once the iteration is complete, the defects are triaged (see below) and the programmers begin repairing the software. When a new build is delivered, the next iteration of testing begins.

After each iteration, the testers create a test report. This is a document (usually a spreadsheet or word processor document) that simply contains a list of all test cases that failed or were not executed. The purpose of the report is to give the project team a good idea of how much of the application was exercised in the test.

For each test case that failed, a tester creates a defect report. These reports are used to determine the general health of the product, in order to allow the project team and stakeholders to understand its maturity and readiness for release (see below about defect tracking).

Testing is complete when either no defects are found or (more likely) all of the defects that have been found satisfy the acceptance criteria in the test plan. These criteria will typically include several rules: that all test cases have been executed, that the project stakeholders have reviewed all of the defects and personally determined that the software can be released with those known defects, and that there is consensus among the engineering team that the product is ready for release. Table 8-6 shows an example of typical acceptance criteria from a test plan.

1. Successful completion of all tasks as documented in the test schedule.

2. Quantity of medium- and low-level defects must be at an acceptable level as determined by the software testing project team lead.

3. User interfaces for all features are functionally complete.

4. Installation documentation and scripts are complete and tested.

5. Development code reviews are complete and all issues addressed. All high-priority issues have been resolved.

6. All outstanding issues pertinent to this release are resolved and closed.

7. All current code must be under source control, must build cleanly, the build process must be automated, and the software components must be labeled with correct version numbers in the version control system.

8. All high-priority defects are corrected and fully tested prior to release.

9. All defects that have not been fixed before release have been reviewed by project stakeholders to confirm that they are acceptable.

10. The end user experience is at an agreed acceptable level.

11. Operational procedures have been written for installation, set up, error recovery, and escalation.

12. There must be no adverse effects on already deployed systems.

The list of acceptance criteria can also include any specific performance requirements ("server must support 80 users," "product must perform under high load for 72 hours without failure," etc.), as well as any security requirements specific to the software.

The majority of the acceptance criteria in the example focus on the consensus of the people involved in the software project. It is important that everyone agrees that any defects that the software ships with will not affect the users.

Note that these criteria assume that there will be defects that will not be fixed. The goal of test execution is not to remove every defect from the software; rather, it is to make the people in the organization aware of the individual issues encountered during testing. This way, everyone is aware of the risks (if any) involved in releasing the software, so an informed decision can be made about whether or not to fix the defects and retest the software.