The Goal: A Successful Test Effort

The goal of testing is a successful test effort, but what is a successful test effort? One that finds bugs? Finding bugs alone does not make a successful test effort. The successful test effort will also ensure that the bugs are removed, and it will probably introduce several improvements.

Note 

The goal of the test effort is to establish a responsive, dependable system in the shortest possible time that satisfies and delights the users while staying within budget, schedule, and resource constraints.

The successful test effort involves several activities in addition to testing the system and finding bugs. Most of these other activities have to do with getting the bugs removed. Testers must perform diagnostics, write bug reports, and track status on the failures that occur during testing; then they must secure, integrate, and verify the fixes. The successful test effort will also complete its test activities in the allotted time. Any plan or estimate that does not take all of these factors into consideration will probably fail, because the failure of any of these activities can cause the entire test effort to fail.

During planning, the potential consequences of a failure in the product and eventually its cost must be balanced against the size and completeness (and cost) of the test effort. Testers need to select a test set that provides the best test coverage for the given time frame. In other words, testers must identify and execute the most important tests.

The tester must be able to communicate the size, complexity, priorities, and underlying assumptions of the test effort during planning, because that is when the schedule is established. If the testers cannot communicate these things clearly and adequately, management will not be able to make well-informed decisions about resource allocations and time frames.

Once testing begins, in addition to producing results reports, testers need to be able to calculate, at any time, how much time is required to complete the planned testing and bug-fix integration activities so they can communicate the impact of events on the schedule and adjust planned activities to control the situation. They need to get the job done in a systematic and reproducible way. This goal of a successful test effort cannot be met without a good plan and measurements.

Management must participate in planning and in building the test agreement. They must actively participate in discussions about what will be tested and what will be fixed, and they must allocate enough of the right resources to get the job done. This means enough time, testers, and test machines to execute the test agreement, and enough programmers to find and fix the bugs within the time allotted for the test cycle.

Management must also recognize that the test scripts are at least as valuable as the source code that they test and they must promote test reuse. This can be accomplished by setting the expectation that test resources will be reused, not only in subsequent releases but as diagnostics and basis suites. Basis and diagnostic test suites should also be shared with other groups.