User Testing vs. Other Testing | Developing User Interfaces for Microsoft Windows

User testing is just one of many types of testing that can occur during the development process. Let's compare user testing to some of the alternatives.

Programmer testing Programmer testing is a cost-effective way to find common user interface errors, some of which are described in the remaining chapters of this book. There are several practical techniques that you can use to find routine usability problems. While I believe that programmers can find usability problems by testing their programs from the user's point of view (by performing real tasks and responding only to what they see on the screen), the problem is that programmers already know how their programs work and sometimes there are usability problems that they simply cannot detect without having a fresh perspective.

Demo testing Demo testing is also a cost-effective way to find common user interface errors and involves giving demos to colleagues, customers, or even random bystanders. The advantage to demo testing is that it forces you to use the program as a user, since when demo testing you typically show how to perform tasks instead of how to use specific features. You are also self-conscious during a demo, so problems that seemed minor during programmer testing tend to seem much worse during a demo. Furthermore, during a demo people typically ask questions and make requests that you never anticipated—you're also able to get a quick reality check to see whether people react to your program in the manner you expect.

QA testing QA testing is a vital part of the testing process, but typically QA is focused on finding bugs and puts less emphasis on more subjective problems like usability. I believe this is wrong—QA testers should be encouraged to find usability problems. Unfortunately, QA testers lose value over time because as they become more familiar with the program they, like programmers, have a difficult time seeing some usability problems.

Beta testing While beta testing is performed with real users, you have no control over what they test and you are not able to witness them using the program. If a beta tester struggles to complete a task, there is no guarantee that you'll get this feedback. Furthermore, if you do get this feedback, you probably won't get specific information that helps you determine why there was a problem. You won't be able to determine what the user was thinking or what specifically the user found confusing.

Each of these types of testing can be effective and provide valuable information. Each has its advantages and plays a strategic role in the development process. But none of them have the critical combination of testing your program with real users and directly observing the testing. This combination makes user testing the most realistic indicator of how users will react to the program. You can't just give your program to people and ask them to go off and use it and then tell you what they think. You have to watch them struggle.

TIP
User testing is the most realistic indicator of how users will react to your program. You have to watch real users struggle to use your program.

If user testing is the most realistic form of testing, why bother with the others? For two reasons. First, user testing finds mostly usability problems and is unlikely to find bugs. The second reason is economic. Although user testing doesn't have to be expensive or time-consuming, user testing is more difficult to do than programmer testing, demo testing, or QA testing. If you can find usability problems by using these other methods, it's more cost-effective to try them first.