14.3 Costs and Benefits of Automated Unit Tests

The main benefit of tests is to prevent and find errors. Errors found during the development phase mean cash money. The removal of an error found during the development is much cheaper than removing the same error when the software has already been delivered. Errors found after delivery lead to unhappy customers, standstill of equipment or business processes, difficult and expensive error localization, and complex problem tracking and remove processes, and eventually to new delivery and installation of the software.

In addition to the enormous savings potential of early error identification, unit tests offer more benefits. Unit tests created by the test-first approach have a strong positive influence on the design. Moreover, unit tests are local, therefore the cause of errors is normally found in the tested class. If an error occurs in the system test, then localizing the error in the entire system is much more expensive. And additional costs (e.g., traveling expenses, standstill of business processes, penalties, etc.) have to be expected if the software is already in productive use.

The benefit of unit tests can be explained by the numbers: Studies conducted by IBM [Subramaniam99] observed that an error found during the coding or unit testing phase is cheaper by a factor of 10, compared to when it is found in the system test. This factor rises to 100 for errors found as late as during the software's productive use.

At the other end of this enormous benefit are the costs for creating and updating tests. These costs are higher for automated than for manual tests. In contrast, the costs for test runs are much higher in manual testing. Figure 14.5 shows that automated tests pay off if they are repeated many times. From a specific number of n iterations, the costs for the n-th manual test run are higher than the costs of the n-th automated test run, plus one-time costs for creating the automated test.

click to expand
Figure 14.5: Comparing the costs of manual versus automated testing.

Another cost factor in automation is the cost for test reworking. When the functionality or the public interface of a class changes, then we also have to adapt our test cases. This means that, in reality, the cost for test automation is not constant; it depends on the (hard to foresee) rate in which the software changes.

Despite the difficulties in estimating the exact costs, we can state the following:

In continuous integration (see the end of Section 14.2) automation of most test cases is a must; after all, these tests run several times daily. Chapter 2 showed this by a small example calculation.
In incremental process models without continuous integration, the costs for automation have to be weighed against the costs for the test runs. It is recommended to collect experience by initially automating the most important basic tests, the so-called smoke tests, ^[3] which check basic application cases (e.g., starting and ending) of the software.
In sequential models, the development is followed by a time-limited integration phase. For this reason, unit tests are needed less often, and the benefit of automation becomes questionable. Often, automating unit tests is uneconomic in strongly sequential processes. Nevertheless, automated tests have to be created for critical components that cannot be tested manually, important internal calculation components for example. In such a case, it is recommended to use a testing framework.

^[3]The program is additionally put through a "smoke test," a relatively simple check, to see whether the product starts to smoke when being switched on.