Chapter 9. Testing Systems

• Need to write a test plan? See Defining the System Test Plan.

• Want to know how to write test cases from use cases? See Use Cases as Sources of Test Cases.

• Need to know how the special features of a program affect testing? See Testing Different Types of Systems.

• Want to know how to quantify the quality of testing? See Measuring Test Coverage.

We have reached the point in the development process in which the word testing is used in its popular sense. But even this meaning has changed. System testing usually refers to the testing of a completed application to determine that it provides all of the behaviors required of the application. We will consider a somewhat broader definition that encompasses completed increments that provide some degree of end-user functionality. We have already discussed this level of testing in Chapter 4 for the case when code has not yet been written. In this chapter we assume that executable code is available.

This level of testing has been seen as "throwing it over the wall" to an unrelated group after the development is completed. However, in an incremental development effort, system testing usually refers to successive rounds of testing end-user functionality with extensive feedback and interaction between the development staff and the test team. The test plan and the development plan must coordinate this interaction.

The words "system" and "application" are used as synonyms by some and distinct concepts by others. When used as distinct concepts, system refers to a "complete" environment including the program being developed, that is, the application, and the operating environment including the operating system, virtual machines, Web browsers, and hardware. We will not make the distinction. We will use the two terms synonymously to mean the complete environment.

Originally we said that testing was a search for defects. Well, that was the truth, but not the whole truth. The focus for testing at this point in the development process shifts from a search for defects that lead to program failures (although some will still be found) to a search for defects that are variances between the actual operation of the system and the requirements for the system. Remember that we took this same viewpoint back in Chapter 4 when we tested the system-level analysis and design models. At that time we investigated whether the analysis and design models were a complete, consistent, and correct solution to the problem stated in the requirements. This early round of validation reduces the number of "variance-from-requirements" defects found during the "end-of-development" round of testing.

This testing phase may be a multilevel activity. For software that will be executed on a processor embedded within special purpose hardware, you should first test the software on a simulator of some type before looking at the actual hardware. The scope of the "system" will also change as development progresses. Eventually "system" test can be applied to the integration of several embedded applications into a single system. For example, many modern automobiles have a number of controllers located in assemblies such as the sunroof or the automatic windows. These controllers communicate with each other in some cases and receive data from sensors in places where there are no controllers.

There are several other types of testing that occur at this point in the development process. These are related directly or indirectly to the requirements. Performance, security, load, and deployment testing activities are all speciality tests conducted under certain circumstances. We will consider these activities in the context of object-oriented systems in this chapter.

Many development contracts call for an acceptance test that is performed by the customer prior to the development activity that's officially ending. This testing is carried out in the deployment environment at the customer site. The customer performs the test and determines whether the product is performing satisfactorily from her point of view. Acceptance testing is a special kind of system testing.

In this chapter we will continue the work from Chapter 4, only now we will assume that code is available. We will consider strategies for selecting the most effective test cases, and we will consider the effect of different testing strategies on the growth of the reliability of the system. But first we must plan.