Section 6.2. Why Automate Your Tests?

6.2. Why Automate Your Tests?

The motivation for automating tests is pretty simple: the easier it is to run the tests, the more the tests will be run, and the more tests you will be able to run. Tests that are not run regularly begin to rot and become useless. Automating unit tests and system tests also makes it easier to run them regularly as part of an automation environment (see Section 3.4). As the number of tests and the different things that you want to test grows, automation becomes the only way to run the tests in a reasonable amount of time. Some tests, such as load testing with many hundreds of machines, are hard to imagine wanting to do by hand!

However, despite what the glossy brochures for test automation tools tell you, automated testing will not magically solve all your testing problems. For each test, you need to first decide how much effort it will take to automate it and what the likely payback is. For instance, imagine that a large, long-running system test costs X each time you run it manually, or costs Y for the first time you automate it and then nothing thereafter. If Y is 10 times X, then there is no benefit to automating the test unless you know that you are going to run the test more than 10 times in the life of the product.

Tests with more human interaction (such as conformance tests, customer tests, and "bakeoffs") benefit less from automation, so this section doesn't discuss automating those kinds of tests.

As a general guideline, all the developers on a project should be able to build and run the unit tests, and there should be an agreed-on set of unit tests to be run whenever developers want assurance that their changes haven't broken anythingthe CYA process.^[1] Some projects in fact require that unit tests are run before committing any changes to the project. There is a balance to be struck here for each project between the time spent running the unit tests and the increased perceived stability of the product. Running the unit tests automatically elsewhere and then mailing failures to developers who have changed the code since the unit tests last passed is another way to approach this issue.

^[1] CYAcover your assets. This can involve anything from a "How Not to Break the Build" document, to a script that must be run before committing changes, to having all your diffs reviewed by other developers.

System tests are often a larger collection of tests that are more complex than unit tests; thus system tests may take much longer to run. Whenever a release is created for testing purposes, the system tests should be run automatically, ideally before any time is spent manually installing and testing the new release. These system tests can be viewed as a contract between developers and testers about what has to work, at a minimum, before testers bother to look at the rest of the product.

Automation of both unit and system tests will also help greatly when the product has to be maintained later on, after the first release. The fewer steps it takes to build and run the tests, the fewer steps there are to forget or to run incorrectly.