Dimensions of Testing

To assess product quality, different kinds of test, each one with a different focus, are needed. These tests can be categorized in various ways:

• Quality dimension: the test focuses on a major quality characteristic or attribute.

• Level of testing: the test strategy targets a general category of software elements.

• Type of test: an individual test has a specific test objective, usually limited to a single quality dimension.

Quality Dimension

There are patterns to quality problems; the same kinds of problems tend to recur in virtually every system. There are various conventions for enumerating quality problems, each designed to help you reason the most important dimension of quality in your context. The following aspects are generally assessed for most products:

• Reliability

  The software performs predictably and consistently; it is resistant to failure during execution: no crashing, hanging, memory leaks, and so on.

• Functionality

  The software executes the required use cases or desired behavior as intended.

• Performance

  The software and the system execute and respond in a timely manner and continue to perform acceptably when subjected to real-world operational characteristics (e.g., load, stress, and lengthy periods of operation). Performance testing focuses on ensuring that the functionality of the system can be provided while satisfying the nonfunctional requirements of the system.

• Usability

  The software is suitable for use by its end users; the focus here is on human factors, esthetics, and so forth.

For each quality dimension, one or more types of test should be executed at different test levels. Additionally, there are other qualities to consider whose assessment may be more subjective : supportability, maintainability, extensibility, flexibility, and so on. Whenever possible, quantitative measures should be derived to assess these qualities.

Levels of Testing

Testing is not a single activity that is executed all at once. In each iteration, testing is executed against different types of targets (targets-of-test) in one or more test cycles. In each test cycle, different targets will be tested , ranging from small elements of the system, such as components (unit testing), to completed systems (system testing). The four levels we use have the following purposes:

• Unit testing

  The smallest testable elements of the system are tested, typically at the same time that those elements are implemented.

• Integration testing

  The integrated units (or components or subsystems) are tested.

• System testing

  The complete application and system (one or more applications) are tested.

• Acceptance testing

  The complete application (or system) is tested by end users (or representatives) to determine readiness for deployment.

Keep in mind that these levels occur throughout the lifecycle, with varying emphasis, either in sequence or in parallel. An early conceptual prototype used in the inception phase to assess the viability of the product vision will be subjected to acceptance tests, informal though they may be. For example, an architectural prototype developed during the elaboration phase will be subjected to integration and system tests to validate architectural integrity and performance of key architectural elements, even though most of the code in the system exists only as stubs. These levels are not "phases" that occur in sequence near the end of the project ”in the iterative lifecycle, testing occurs early and often.

Types of Test

There are many types of test, each one focusing on a specific test objective, testing a characteristic or attribute of the software, or finding certain classes of faults or failures. Because tests are executed throughout the lifecycle, the software being tested may be a single unit of code, integrated units, or a complete application or system. Here are some of the most common types of test:

• Benchmark test

  Assesses the performance of a target-of-test against a known standard such as existing software or measurement(s)

• Configuration test

  Assesses how the target-of-test functions on different test configurations (hardware or software)

• Functional test

  Assesses how the target-of-test functions execute the required use case as intended

• Installation test

  Assesses how the target-of-test installs on different configurations or under different conditions, such as insufficient disk space

• Integrity test

  Assesses the target-of-test's reliability, robustness, and resistance to failure during execution

• Load test

  Assesses the acceptability of the target-of test's performance under varying operational conditions, such as number of users, number of transactions, and so on, while the configuration remains constant

• Performance test

  Assesses the acceptability of the target-of-test's performance using various configurations while the operational conditions remain constant

• Stress test

  Assesses the target-of-test's performance when abnormal or extreme conditions are encountered , such as diminished resources or an extremely high number of users

Regression Testing

Regression testing is a test strategy in which previously executed tests are reexecuted against a new version of the target-of-test to ensure that the quality of the target has not regressed (moved backward) when new capabilities have been added. The purpose of the regression test is to ensure two things:

• The defects identified in the earlier execution of test have been addressed.

• The changes made to the code have not introduced new defects or reintroduced old ones.

Regression testing can involve the reexecution of any of the test types. Typically, some regression testing is performed during each iteration, rerunning tests from prior iterations, and therefore is of particular importance in our lifecycle model.