The Importance of Independent Testers

After reading the previous section, you might conclude that as long as your customers work with the product, and if you've tested the product continuously throughout the development during each iteration, you should be okay. All of this testing helps, and we believe that it is necessary. But you still need independent testers to test your software ”as soon as possible!

Fortunately for us, Raj joined us at the end of the Construction phase, and he made an immediate impact. He uncovered defects that we might never have caught. He also automated the acceptance tests so we could easily determine when we were ready to deploy the software to the customer. More importantly, he provided an independent voice of reason for our group . (Of course, now we wish he had joined us much earlier in the project.)

The Brown Thumb

Some organizations mistakenly think that testing is all about putting quality into the code ”at the end of the project! This is an impossible task. You have to build quality into the product from the start of the project. Testing is about verifying what the developers should already know (that the software works as specified), ensuring that integration issues are addressed, and about uncovering defects that might not be obvious. In other words, testing is about trying to break the software.

What does it take to be a good tester? One of Gary's teachers said it succinctly: A good tester has a "brown thumb." This means that a good tester can take software that people think is working and turn it to crap quickly. ^[2] The good tester uncovers huge numbers of meaningful defects in a short time.

^[2] The term came from Dr. Bill McKeeman at the Wang Institute of Graduate Studies. The brown thumb is the opposite of a green thumb. People with green thumbs have an uncanny ability to grow all kinds of plants, seemingly with ease, while those lacking the green thumb either grow plants with difficulty, kill the plants, or are most expert at creating compost.

Soon after Raj joined our team, we knew that he was gifted with a brown thumb. Gary used to have a brown thumb, but because he was out of practice, his thumb had become several shades lighter.

Testing Your Own Software

You might ask whether you can test software that you have also written, to determine whether the software is good enough to ship. In general, the answer is no. We know of only rare cases where this is possible. However, several factors can contribute to your ability to assist with testing your own software.

Define Requirements

The requirements for the software you are building must be very well-defined . If you are going to do your own testing, or if your customer is paying you to deliver a specific feature set, there must be little or no doubt about what it means to satisfy requirements and meet customer expectations. The best examples we have seen of such software are compilers for programming languages. Most languages are formally specified and there is often an acceptance test suite for a given language. ^[3] Software that contains a considerable amount of graphical user interface is much more difficult to define unambiguously than a language processing program such as a compiler.

^[3] Passing all tests in a test suite doesn't guarantee that you have implemented a defect-free compiler for the language, but it provides a minimum set of tests that you must pass to be compliant with the language.

Have the Right Mindset

You need to have the right mindset. What does it mean to have the right mindset? There are people who firmly believe that they can produce defect-free software. Whether they are able to do so is not the issue. If you don't believe you can do something, you might as well not try. If you do believe that you can deliver defect-free code to your customer, then you must be willing to do whatever is necessary to make that happen.

Organizations can also develop the right mindset. The following example is a short side-trip that illustrates what we mean.

In 1984, Gary had the privilege of working at the Datapoint Technology Center (DATEC) in San Antonio, Texas. At the time, Datapoint was a leader in innovative computer technology. Although the technology center was a research group, its members took tremendous pride in delivering high-quality software to the rest of the company. Gary joined and worked on a compiler code generator for the company's programming language. He had never worked on a code generator up to that point, and he was about to build one from scratch. There were code generators he could study and colleagues he could consult with, but the responsibility to design and deliver the code was his. ^[4]

^[4] A code generator may not be specified quite as formally as the front-end of a compiler, but it is still fairly easy to determine if generated code is correct or not.

DATEC had an interesting tradition. When a staff member said that code was ready for release to the user community, there was an implicit agreement that accompanied the code. The first person to discover and report each bug in the code received a free lunch at the expense of the staff member who released the code. This can be quite daunting, and very frightening. All Gary could think of was, "There goes the mortgage payment. There goes the house. My marriage is ruined. "

There were only three choices ”never release the code generator, release it and hope there weren't too many defects in it, or do everything possible to make it bulletproof. Gary chose this last course of action. When he released the code generator, only three defects were discovered during the next year. Certainly, this was affordable. What is more amazing is that the three defects in a year was about the average for the group. The whole group had the right mindset.

Beware of the Hawthorne Effect

Sometimes you may think you are improving your work, but the improvement may only be temporary. For example, you need to know how to write good tests in order to improve your testing. We often overlook this need and think that as long as we perform a testing activity, such as unit testing, we will magically attain the hoped-for results of high-quality software.

Sometimes there is a benefit to just "going through the motions ." For example, in a class on empirical software engineering, Gary's students conducted an experiment on Test-First Programming (TFP), which is a version of the XP testing practice (see Appendix C). A group of programmers were divided into two subgroups and given the same program specification. One group was asked to write their tests first and the other was asked to write the program using their normal (not test-first) approach. The results, while statistically inconclusive, indicated that TFP might be a valuable tool for producing higher quality code. ^[5] An interesting observation was that the tests written by most of the TFP group weren't good tests. The training provided for the participants was minimal. Yet, the TFP team seemed to produce better code. A possible explanation is that simply taking the time to think about tests, regardless of whether the tests were well-designed, decreased the number of defects found in the code. These results may simply be a manifestation of the Hawthorne Effect. ^[6] If so, the benefits realized from this study would eventually revert to earlier levels.

^[5] For a result to be statistically significant, you need to have a 95% confidence that your prediction is correct. Our data, using a small sample size, indicated an 89% confidence level. We believe the results warrant more experimentation.

^[6] The Hawthorne Effect was discovered in a study of workers at the Western Electric plant in Hawthorne, New York. Production increased, not as a result of actual changes in working conditions, but simply by the fact that the work was being observed .

The act of doing TFP might only provide a short-term increase in software quality, but lasting improvement requires more effort. If TFP is really useful in the long term, then it will only be useful if you write good tests. You need to learn how to write good tests, and then internalize that knowledge so that it becomes a natural part of your work. It takes time and perseverance to make any new skill a habit.

If you are lucky to have excellent requirements, the right mindset, and the ability to write good tests, you have a chance ”albeit a slim chance ”of testing your own work. On a small project, like PSP Tools, we thought it might work. It did not.

We should have realized sooner that we needed an independent tester, and then invited one to join us much earlier in the project. We were lucky that Raj was able to join us when he did. His participation and contributions helped us deliver better code.