Test documentation

10.3 Test documentation

Test documentation includes all test program documents from the overall test plan through the final test report. Test documentation is a parallel effort, as shown in Figure 10.1. It starts with the original requirements statement, like the development documentation. On the basis of the requirements, test plans, cases, scenarios, procedures, data, and results documentation are generated as the SDLC progresses. (Chapter 4 addresses the topic of test documentation more fully.) Testing is documented through a series of increasingly specific documents, starting with the test plan and including test cases, test scenarios, detailed test procedures, test data, and test results. Test documentation spells out the sequence of events by which compliance of the software with its requirements is ultimately demonstrated.

Figure 10.1: Software test development.

Software quality practitioners play a major role in the whole testing process. They may, in fact, actually conduct the testing at the acceptance level. Nonetheless, the software quality practitioner must carefully review the entire test program to make sure it is sufficient to exercise the software in a manner that will maximize the defect-finding capability of the tests. Remember that the goal of testing is to find defects. Software quality practitioners must be sure that this goal is being met. The second goal is to demonstrate that the software performs as the approved requirements demand. Software quality practitioners, together with the user or customer, must review the testing and ascertain whether the software does perform as required. When deficiencies are found, either in the tests or in the results, the software quality practitioner is responsible for making sure that the deficiencies are recognized by management so that appropriate action is taken.

10.3.1 Test plan

As shown in Figure 10.1 (and in Figure 4.3), test documentation begins with the test plan (see Appendix G), which is based on the original requirements. In fact, the initial test planning is performed during the requirements phase. This not only gets the test activities off to an early start but helps to ensure the measurability and testability of the requirements themselves. The test plan will grow and evolve, just as the requirements grow and evolve, throughout the development life cycle, but it is important to begin at this point in order to keep pace with the development of the code. Plans are made for the expected test tools, data generators, simulators, and so on that are anticipated to be needed.

10.3.2 Test cases

As the design begins to mature and functions are identified, test cases (see Appendix H) are correspondingly identified. These groups of individual tests will be applied to major sections of the software. They are usually based on logical groupings of requirements in much the same way as the functional design is approached. If Use Cases were used in the requirements, they are logical bases for the test cases. Test scenarios are optional subsets of the test cases. They provide for the simplification of complicated or lengthy test cases.

10.3.3 Test data

Initial test data requirements are also identified at this time. Test data must be provided, not to show that the software works as it was written, but that it works as was intended by the requirements. This means that the test data must cover as wide a spectrum of both legal and illegal values as possible. Nominal values, as prescribed in the various design documentation, will only show that the software meets nominal conditions. The object of testing is to uncover defects in the software, so the test data must be carefully chosen to present abnormal and incorrect inputs as well as expected inputs to determine the response of the software to unexpected, borderline, and erroneous conditions.

10.3.4 Test procedures

As the design and coding progress, test procedures (see Figure 4.4) are prepared. Test procedures are the step-by-step actions to be taken during each test. Every operator action, every data entry, and every expected response is specified in a sequence of steps for the given test. In that way, the exact conditions of the test are controlled, and each actual output or response of the software can be compared against the expected result. Each difference between the expected and actual results is recorded as a probable defect and is analyzed to determine whether or not the software is performing as required.

10.3.5 Test reports

The test reports (see Appendix I) record the actual happenings during each test. They specify the expected and actual results and the conclusions drawn from the results. Anomalies and the final disposition of the anomalies are recorded. Test reports are the key factor in determining when testing has reached its beneficial conclusion.