Section 6.1. Different Kinds of Tests

6.1. Different Kinds of Tests

There are many different ways to test software products. The tests that are described in this section are all tests that are separate from the productthat is, tests that don't have to be shipped to a customer. Other kinds of tests not discussed here are written directly into the product, checking for sane data values at the start of a function or testing for consistent states at other places in the source code.

All tests contain assumptions about what is correct and incorrect behavior for the product. Few products are ever mathematically specified to their most precise details, so most test results can only be as accurate as their assumptions are about how the product should work. Tests that check for both known correct and known incorrect behavior at least catch each end of the range of a product's behavior. The area between these two extremes is where carefully constructed tests can detect any poorly specified or inconsistent parts of a product, in addition to errors in the implementation of the product.

6.1.1. Unit Tests

Unit tests are designed to test smaller parts of a product, ideally just a single part per unit test. Exactly what a "single part" means in practice depends on how the product is designed: it could be a whole feature, a class, or just a single method or function. Unit tests are often written by the same developers who wrote the source code that is being tested. They may in fact be the best source of documentation of how to write applications that use their particular part of the product. Unit tests generally don't make assumptions that a network is accessible and working, or that a database or any other large product is available.

Unit tests are closely tied to the code that they test, and indeed the unit tests for the functions in a file foo.c will often be found in a file named foo_test.c in the same directory. The ability to identify source files for tests by their names is often useful, so don't just put unit test code anywhere. Unit tests are useful only if they stay in step with what they are testing, so it's good practice to build the unit tests whenever the product is built. The extra time for a build is a one-off cost, since a good build tool will rebuild unit tests only when something changes. Sometimes unit test files are placed in a top-level directory parallel to the product source code, but this doesn't encourage checking out, compiling, and running the tests, so I don't recommend doing this. Separate top-level directories can sometimes make running the preprocessor more complicated for C and C++ test files, but they may also make packaging Java class files a little easier.

Some programming methodologies such as XP stress that unit tests should be written before the code that they will test is written. This kind of test-driven development (TDD) approach does give a clearer understanding of what the code should do and what its APIs should look like. Of course, this also involves more work in the short term, and testing every get and set method may be unnecessary. As with all testing, the art of writing unit tests lies in deciding where to test for the maximum likely reward for your effort.

Regression tests are defined as any tests that confirm that a program hasn't regressed and reintroduced bugs. Not regressing has also come to mean that a program still produces the same test results as it did before some changes were made to the program. Using regression tests can reduce the risk of making changes.

6.1.2. System Tests

System tests are designed to test the whole product. They may well be written by testers, often a separate group of people from the developers who wrote the source code for the product. Having two groups of people provides a useful confirmation when both groups have the same understanding about how the product is supposed to work. Testers who write system tests may or may not know how the product works at the source code level, so these kinds of tests are sometimes referred to as "black box testing." Testing that depends upon being able to see inside some imaginary box is known as "white box testing."

System tests generally assume that the product has been installed as the customer will install it and that everything that the product requires is present. This may include a working connection to the Internet or access to a database. Specifying exactly what a product requires to operate correctly is discussed in Section 9.2.1.

6.1.3. Customer Tests

Customer tests involve using the product just as a customer might. This might include installing the product in different ways or updating the product to a newer version. Customer testing may involve beta releases with real customers. Usability tests, which try to decide how easy a product is to use, are often conducted as part of customer tests. Both novice and experienced users of the product are needed for usability tests to be able to improve the design of a product.

6.1.4. Other Tests

Other kinds of tests include:

Comparison tests

Similar products are compared directly against each other, often in terms of features, robustness, and price.

Conformance tests

Testing that a product conforms to a particular specification.

Interoperability tests

Products are tested for how well they work together, usually by assuming that they all conform to a particular specification and then connecting the different products together in some way.

Many of these kinds of tests are performed by an external organization on behalf of a company, a standards body, or an industry magazine. Some of these organizations are more rigorous and objective than others, just like the industry analysts who write product reports for their magazines.

Such testing organizations may permit support staff from a company to be present but will not usually allow them to actually configure their own product except by proxy. It's also hard for testing organizations to be sure that the product they are testing is anything like the version that a customer reading the test results six months later will be able to purchase.

The design of these kinds of tests is often a matter of much discussion and careful market maneuvering. If you can influence the test configuration and parameters, then you can help ensure that your product will succeed, even before the test is performed. Indeed, products are sometimes specifically designed to perform well in benchmark tests, even if their performance in more common, real-life situations is less spectacular.