Types of testing

4.1 Types of testing

The four most common types of testing are unit, module, integration, and user or acceptance testing. The tests may have different names in different organizations, but the types are basically the same.

4.1.1 Unit testing

Unit testing, the most basic type of software testing, ignores the concept that document reviews are really a type of testing. Unit testing is usually conducted by the individual producer.

Unit testing is primarily a debugging activity that concentrates on the removal of coding mistakes. It is part and parcel with the coding activity itself. Even though unit testing is conducted almost as a part of the day-today development activity, there must be some level of planning for it. The programmer should have documented at least the test data and cases he or she plans to use and the results expected from each test. Part of each walkthrough or inspection of the software should be dedicated to the review of the unit test plans so that peers can be sure that the programmer has given thought to the test needs at that level.

It is worth reiterating the tenet that the tests run must be oriented to the finding of defects in the software, not showing that the software runs as it is written. Further, the defects being found will include not only mistakes in the coding of the unit, but design and even requirements inadequacies or outright mistakes. Even though the unit is the smallest individually compilable portion of the software system, its interfaces and data manipulation can point out wide-reaching defects. Informal though it may be, the unit testing activity is the first chance to see some of the software in action.

It can be seen that the rule of finding defects, not showing that software runs, could be in jeopardy here. In fact, a tradeoff is in play with having programmers test their own software. The expense, in both time and personnel, to introduce an independent tester at this point usually offsets the danger of inadequate testing. With high-quality peer reviews and good, though informal, documentation of the tests and their results, the risk is reduced to a low level. Software quality practitioners, in their audits of the UDF and their reviews of the testing program as a whole, will also give close attention to the unit test plans and results.

4.1.2 Module testing

Module testing is a combination of debugging and integration. It is sometimes called glass box testing (or white box testing), because the tester has good visibility into the structure of the software and frequently has access to the actual source code with which to develop the test strategies. As integration proceeds, the visibility into the actual code is diminished and the term gray box is heard. This refers to the more limited visibility into the software and its structure.

As units are integrated into their respective modules, the testing moves appropriately from a unit testing-debugging-mode into the more rigorous module testing mode. Module integration and testing examine the functional entities of the system. Each module is assigned some specific function of the software system to perform. As the units that comprise the module are brought together into that functional unit, the module tests are run.

The testing program becomes somewhat more rigorous at the module level since the individual programmer is not now the primary tester. There will be in place a more detailed test plan, sets of data and test cases, and expected results.

The recording of defects is also more comprehensive at this stage of the test program. Defects are recorded in defect history logs and regularized test reports are prepared. As they are found, the defects are fed back into the code and unit test phase for correction. Each defect is tracked from its finding and reporting through its correction and retest. The results of the correction are monitored and controlled by the CM system that is begun at this point in the SLC. This is important, since many of the errors that have been made, and defects that have been discovered, will affect the design and requirements documentation.

Most of the minor coding mistakes will have been caught and corrected in the unit testing process. The defects that are being found in the module tests are more global in nature, tending to affect multiple units and modules. Defects in interfaces and data structures are common, but a significant number of the defects will involve deficiencies in the design and requirements. As these deficiencies come to light and are corrected, the design and requirements baselines will change.

It is critical to the rest of the SLC that close control of the evolving documents be maintained. If the corrections to the defects found in the test program are allowed to change the products of earlier SLC phases without proper control and documentation, the software system can quickly get out of control. When requirements or design change without commensurate modification to the rest of the system, there will come a time when the various pieces do not fit together, and it will not be clear which versions of the units and modules are correct.

Software quality practitioners will have reviewed the test plans, and the rest of the documentation, prior to the conduct of the module tests. Software quality practitioners are also expected to review the results of the testing. These reviews provide assurance that defects will be recorded, tracked, resolved, and configuration-managed.

4.1.3 Integration testing

Integration testing may be considered to have officially begun when the modules begin to be tested together. This type of testing, sometimes called gray box as discussed in Section 4.1.2, implies a limited visibility into the software and its structure. As integration proceeds, gray box testing approaches black box testing, which is more nearly pure function testing, with no reliance on knowledge of software structure or software itself. Integration testing is largely guided by the critical or detailed design and then the preliminary or architectural design.

As modules pass their individual tests, they are brought together into functional groups and tested. Testing of the integrated modules is designed to find latent defects, as well as interface and database defects. Because the testing up to this point has been of individual modules, several types of defects cannot be detected. Such things as database interference, timing conflicts, interface mismatches, memory overlaps, and so on are found only when modules are forced to work together in integrated packages.

Integration testing uses the same sorts of conditions and data as the individual module tests. Valid data and messages are input, as are invalid conditions and situations. The test designer must be very creative in seeking out valid combinations of circumstances that are possible, but may provide illegal or invalid conditions to the software. The response of the integrated software to these situations is determined, as well as the software performance with valid inputs.

Integration testing is the level at which the quality control practitioner or tester begins to see differences between traditional systems and client-server or distributed processing applications. The greatly increased sets of inputs and initial conditions require some of the more elaborate testing schemes such as record and playback, automated test generation, software characterization, data equivalence, sampling, and the like.

The reporting of test results is very important in the integration test period. How the software responds is recorded and analyzed so that corrections can be made that fix the experienced defect but do not introduce new defects somewhere else. Error and defect logs should be maintained for trend analysis that can point to particularly vulnerable portions of the software and its development. These portions can then receive additional testing to ferret out deep-seated anomalies and improper responses. Close control must be maintained of the configuration of the software system through this period so that all changes are properly documented and tracked. It is in this time frame that many software systems get out of hand and accounting is lost as to which version of which unit, module, or subsystem are the proper one to use at any point.

It is the integration test phase that will uncover many hidden defects with the design and requirements. Formal reviews and less-formal walkthroughs and inspections have been used to find many of the design and requirements defects. But as the software is put into use in an increasingly realistic manner, other defects may surface that were beyond the depth of the earlier defect-finding efforts. As defects are found in design or requirements, they must be corrected and the changes in the earlier documents made. This, in turn, may necessitate rework of design, code, and earlier testing. Finding such serious defects at this point is very expensive but less so than finding the defects in the operations phase. Thus, every effort must be made to maximize the defect-finding capability of the integration tests.

An important role for the software quality practitioner in this effort is the review of the integration test plans, cases, scenarios, and procedures. Software quality practitioners should make every effort to assure that the integration tests cover the full range of capabilities of the integrated set of modules. Review of the test results, and the decisions made on the basis of those results, should also be reviewed and approved by the software quality practitioner before testing progresses beyond the integration phase.

4.1.4 User or acceptance testing

User testing is primarily intended to demonstrate that the software complies with its requirements. This type of testing is black box testing, which does not rely on knowledge of the software or the structure of the software. This testing is intended to challenge the software in relation to its satisfaction of the functional requirements.

These tests have been planned based on the requirements as approved by the user or customer. All testing up to this time has been oriented to finding defects in the software. The early tests were also based on the requirements but were designed to show that the software did not comply in one fashion or another to the requirements. By the time the acceptance-testing period is reached, the software should be in a sufficiently defect-free state to permit the emphasis to change.

One important aspect of the acceptance test is that, whenever possible, it is performed by actual intended users of the system. In that way, while it is being shown that the software complies with its requirements, there is still the opportunity to introduce anomalous user actions that have not as yet been encountered. Persons unfamiliar with the system may enter data in incorrect, though technically permitted, ways. They may push the wrong buttons or the correct buttons in an incorrect sequence. The response to these unexpected or incorrect situations is of great importance to users who do not want their system to collapse due to human mistakes. The overriding requirement for every system is that it perform its intended function. This means that if incorrect actions or data are presented, the system will not just abort but will tell the user what has been done wrong and will provide an opportunity for the user to retry the action or input. Invalid data received from outside sources should also be treated in such a manner as to prevent collapse of the system.

Another important consideration of an acceptance test is verification that the new software does not cause changes to workflow or user responsibilities that have been overlooked. While it may be shown that the software performs exactly as expected, the associated human-factor changes may make the system difficult to use or cause negative effects on the related work of the users.

The acceptance or user test is usually the last step before the user or customer takes possession of the software system. It is important that software quality and CM practitioners play very active roles in reviewing and executing the tests, as well as in the change management of the system during this period. Software quality practitioners may even have performed the full execution of the acceptance test as a dry run prior to the system's release for the user operation of the test. CM of the system at this time is critical to the eventual delivery of the exact system that passes the acceptance test.

4.1.5 Special types of tests

Four types of tests may be considered to fall into the special category. These tests are planned and documented according to the same rules and standards as the other types of tests, but they have specific applications. The four major special tests are as follows:

1. Regression;

2. Stress;

3. Recovery;

4. Back-out and restoration.

4.1.5.1 Regression tests

Regression tests show that modifications to one part of the system have not invalidated some other part. Regression tests are usually a subset of the user or acceptance test. They are maintained for the verification that changes made as a result of defects or enhancements during operation do not result in failures in other parts of the system. Regression tests are an abbreviated revalidation of the entire system using generally valid data to show that the parts that were operating correctly before the changes are still performing as required.

A number of studies indicate that as many as 50% of all changes made to a software system result in the introduction of new defects. This may be low or high, but there is a significant risk involved in the introduction of corrections. Some, of course, are errors in the change being made, such as coding errors, change design mistakes, and the like. Others, however, come from unexpected interactions with subsystems other than the one being modified. A change to the way a database variable is updated in one module may affect the time at which another module should read that variable in its own computations.

Close CM control and analysis of changes, and their impact on the system as a whole, is necessary. Software quality practitioners must be sure that a change control board or equivalent function is involved in all change activity during both integration testing and the operation phases of the SLC. This protects the integrity of the baseline system itself and helps ensure that changes are being made to the correct versions of the affected software. Delivery of the proper versions of the modifications is also a function of CM, which software quality practitioners must monitor.

4.1.5.2 Stress tests

Stress tests cover situations that occur when the software is pushed to, or beyond, its limits of required capability. Such situations as the end of the day, when 23:59:59 hours becomes 00:00:00 and the software is required to recognize that 00:00:00 is later that 23:59:59, must be challenged. The rollover of the year field is also a situation ripe for testing. The well-known Y2K situation was concerned with whether software would realize that the year 00 or 000 was later than the year 99 or 999.

Other stress situations occur when the software is presented with the full number of transactions it is expected to handle plus one or two more. What happens when the transaction n + 1 is presented? Does one of the existing transactions get overwritten? Is there a weighting algorithm that selects some transaction for replacement? Is the new transaction merely ignored?

Still another test case is the situation in which the software is run for a long time without interruption. This could easily expose flaws in housekeeping or initialization routines.

Stress tests are an important part of any test program. The types of stress that might be exercised will become apparent as the software develops and the testers understand its construction more clearly. There should be a valid way of handling these and other situations spelled out in the requirements statement. The compliance of the software with the requirement is to be challenged.

4.1.5.3 Recovery tests

Most data centers have recovery procedures for the repair of data on a damaged disk or tape, and they also consider the case of operator errors that may invalidate some of the data being processed. Recovery from abnormal terminations or system crashes may also require recovery tests.

Recovery testing is conducted when a hardware fault or operating error damages the software or the data. This type of testing is critical to the confidence of the user when a data or software restoration has been conducted.

Often, restoration testing can be accomplished by using the regression test software. In other cases, the full acceptance test might be required to restore confidence in the software and its data.

4.1.5.4 Back-out and restoration tests

To back-out and restore is the decision to remove a new software system in favor of the older version that it replaced. Needless to say, developers are usually very embarrassed by such an action because it recognizes that the new system was insufficiently tested or was still so error-ridden that it was worse to use than the old system.

In the case of back-out and restore, the new system is removed from production, any new database conditions are restored to the way they would have been under the old system, and the old system itself is restarted. In the least critical case, the database used by the new system is the same as that of the old system. More often, though, the new system provides expanded database content as well as improved processing. When the contents of the new database must be condensed back into the form of the old database, care must be taken to restore the data to the form in which the old system would have used it.

The testing required includes at least the acceptance test of the old system, which often is augmented by the running of the most recent set of regression tests used for the old system. Clearly, there must have been some planning for back-out and replacement when the new system was installed. The old system would normally have been archived, but the saving of the acceptance test and the regression tests must also have been part of the archiving process.

Great care must be taken to see that the backing out and replacement are well planned and carried out. Time-critical issues regarding the need for the replaced system can add to the concerns and complexity of this action.

It is rare that a newly installed system is so badly flawed that it must be replaced. However, it is the responsibility of the quality practitioner to make management aware of the threat, no matter how remote.