User Testing Procedure

Once you have set your goals and selected test users, you're ready to start the process. The following procedure gives the basic steps for performing user testing on the cheap.

Prepare a Test Plan

You first need to prepare a test plan, which needs to address the following issues:

• Select the tasks that the test users will perform Again, these tasks should be essential to the basic functionality of the program or involve features that have unusual or complex user interfaces. Note that you need to test tasks, not features, since it is meaningful to ask a user to perform a task but it doesn't make any sense to ask a user to use a feature. If your goal is to test specific features, test tasks that incorporate those features. Try to select realistic tasks that are likely to happen in the real world rather than contrived tasks designed solely to test specific features.
• Determine whether the Help system is part of the test If a test user is confused, he might try to use the online Help. This should be part of the test only if the Help system is developed enough to actually be helpful. You don't need to perform user testing to conclude that poorly developed, incomplete Help isn't helpful.
• Determine the length of each test While some user tests can take several hours, I recommend keeping most tests to under an hour. Two hours should be the maximum. If a test tasks more than two hours, you should divide it into parts and have different users test different parts. Each test user doesn't have to do every test.
• Determine the number of test users required Again, three test users are usually the minimum.
• Determine the equipment and materials you need to perform the test You'll need a furnished room, a computer (except for paper prototypes), and the program you want to test. You'll certainly need a notebook to take notes. You might choose to use a video camera or an audio recorder to record the test. You might also want to prepare a questionnaire for the test users to fill out after the testing is finished.
• Determine who will run the tests As I'll discuss later in this chapter, it may not be a good idea to have the programmers responsible for the user interface run the tests—they might not be receptive to bad news.
• Schedule the users for the test Select test users, and make sure that their background matches the program's target users as much as you can. Be sure to document their background, since this information will help you evaluate the test results. For scheduling, be sure to leave enough time between each test for the test user instructions and wrap-up, each of which I'll describe later.

Consider a Test Run

If you plan to do a lot of testing, you should consider doing a test run. You need to make sure that the test makes sense and that the selected tasks accomplish what you want them to. You need to work out any problems beforehand.

Now you are ready to test with the real test users.

Give the Test User Instructions

Before you start testing, give the test user instructions about the test. These instructions are very important, as they can significantly affect the results of your tests. They will help you get as much information out of the testing process as possible. Take these steps before beginning user testing:

• Introduce yourself
• Introduce the program that the user is going to test Explain what the program is for. You should explain the state of the program, that is, whether the test user is testing a prototype, a nearly finished program, or something in between. But regardless of the program state, you need to make it clear that everything is changeable and that you would be happy to throw away what isn't good. The test user should not feel reluctant to make suggestions because the program appears to be in an advanced state.
• Explain that you are looking for problems in the software You are especially looking for problems where the program is difficult to use. The test user shouldn't feel bad if he can't complete a task. That's the program's fault, not the test user's, and that is the kind of information you are looking for. The goal is to find problems, not to complete tasks. Make it clear that you are testing the software, not the test user. Lastly, explain that since the goal is to find problems, the test user won't hurt anyone's feelings by criticizing the program.
• Give an overview of the testing process Explain what tasks the test user is to perform and roughly how long it will take.
• Explain that the test user should not worry about harming the computer Test users should use the computer exactly as if it were theirs. For example, if a task requires saving a file, the user should go ahead and save the file. Test users shouldn't worry about harming the computer or leaving test files. Also, the test user should feel free to make any adjustments to the monitor, keyboard, mouse, chair, or anything else to be more comfortable. Obviously, you should select a computer for testing that the test user really can't harm. You probably shouldn't use your development computer. This step obviously doesn't apply to paper prototypes.
• Have the test user sign any forms You might choose to have the test user sign a consent form, as I describe later. If the program you are testing reveals anything you don't want made public, you should also have the test user sign a nondisclosure agreement.
• Explain that participation in the test is voluntary and the test user can leave at any time Explain to the test user that if the test makes him feel uncomfortable or if he prefers not to continue, he can stop the test for any reason. The last instruction is especially interesting. Although it might seem as if asking a test user to try a couple tasks with a computer program is no big deal, there have been some tests performed on people that are now considered highly unethical if not illegal. The most famous of such tests is Stanley Milgram's "Obedience to Authority" tests conducted in the late 1950s. In these tests, a test subject was told that he, "the teacher," and another subject, "the learner," were going to perform a test on learning behavior. The subject was to ask the learner a series of questions involving word pairs. If the learner missed a question, the teacher was to deliver an electric shock, starting at 15 volts and going all the way up to 450 volts in 15-volt increments, and repeat the question until the learner got it right. As the learner missed more and more questions, the teacher would apply ever-increasing jolts and the learner would beg the teacher to stop. Only a few did. In reality, the learner was really an actor and there weren't really any electric shocks applied. The point of the test wasn't to test learning behavior but to test the participant's willingness to follow the instructions of an authority figure, even to the harm of others.

There are now laws on the books as a result of such testing. While I know you would never apply electric shocks to your test users, you need to understand that user testing is serious business. Some users find the testing process to be very emotional, especially if they are not able to perform the tasks. Some companies require test users to sign a consent form that states that the test user has agreed to participate and that gives you permission to use the test results, including any audio tapes or videotapes. You might want to ask your company's legal counsel for advice on this matter.

Ask the Test User to Think Out Loud

You should now ask the test user to think out loud. This is the most important instruction you can give. Thinking out loud is when the test user verbalizes all his thoughts while performing a task. Having the test user think out loud helps you understand exactly what the user is doing and thinking. It helps you know if the user is confused or has made incorrect assumptions. Most importantly, it helps you understand why. Having the test user think out loud helps you get into the test user's head. You don't need to guess what the user is thinking.

TIP

---

Ask the test user to think out loud, that is, to verbalize all thoughts while performing a task.

Since most people are unfamiliar with this concept, explain to the test user what thinking out loud is. You should then demonstrate how to think out loud by performing a simple task, such as changing the desktop's wallpaper. This example will make what is a fairly abstract concept easier to understand. You can use the following instructions: "We find the tests are more helpful if the participants say what they are thinking while they are using the program. Just say what you are thinking—whatever comes to mind. For example, if you are thinking about clicking a button, say why you think you need to click that button and what you expect to happen once you click it. To help you understand what I have in mind, let me give you a quick demonstration…" Some test users find thinking out loud difficult to do. If the test user starts staring quietly at the screen for a while, just ask "So, what are you thinking now?" to get the test user going again.

The only time I would not recommend asking the test user to think out loud is when it undermines what you are trying to test. For example, if you're trying to test how efficient an order entry system is by timing how long it takes to perform the task several times, thinking out loud would make any time measurements less accurate since it forces the user to work at the speed at which he can talk or find the words to describe what he is thinking.

Avoid Giving the Test User Assistance

Another important instruction you need to give is that you will only be watching the test user perform the tasks and will not be able to answer any questions about performing those tasks. You should explain that it is important to make the test as realistic as possible and real users do not have expert users standing by to answer questions. The goal of the test is to determine the user's ability to use the software, and this goal will be undermined if you tell the user how to do it. However, since you want the test user to think out loud, explain that you want the test user to ask any question she may have—you just won't be able to answer them. You should also tell the test user that you would be happy to answer any questions once the test has completed.

TIP

---

Avoid giving the test user assistance during the test. Encourage the test user to ask any questions—you just won't be able to answer them.