Once you have set your goals and selected test users, you're ready to start the process. The following procedure gives the basic steps for performing user testing on the cheap.
You first need to prepare a test plan, which needs to address the following issues:
If you plan to do a lot of testing, you should consider doing a test run. You need to make sure that the test makes sense and that the selected tasks accomplish what you want them to. You need to work out any problems beforehand.
Now you are ready to test with the real test users.
Before you start testing, give the test user instructions about the test. These instructions are very important, as they can significantly affect the results of your tests. They will help you get as much information out of the testing process as possible. Take these steps before beginning user testing:
There are now laws on the books as a result of such testing. While I know you would never apply electric shocks to your test users, you need to understand that user testing is serious business. Some users find the testing process to be very emotional, especially if they are not able to perform the tasks. Some companies require test users to sign a consent form that states that the test user has agreed to participate and that gives you permission to use the test results, including any audio tapes or videotapes. You might want to ask your company's legal counsel for advice on this matter.
You should now ask the test user to think out loud. This is the most important instruction you can give. Thinking out loud is when the test user verbalizes all his thoughts while performing a task. Having the test user think out loud helps you understand exactly what the user is doing and thinking. It helps you know if the user is confused or has made incorrect assumptions. Most importantly, it helps you understand why. Having the test user think out loud helps you get into the test user's head. You don't need to guess what the user is thinking.
TIP
Ask the test user to think out loud, that is, to verbalize all thoughts while performing a task.
Since most people are unfamiliar with this concept, explain to the test user what thinking out loud is. You should then demonstrate how to think out loud by performing a simple task, such as changing the desktop's wallpaper. This example will make what is a fairly abstract concept easier to understand. You can use the following instructions: "We find the tests are more helpful if the participants say what they are thinking while they are using the program. Just say what you are thinking—whatever comes to mind. For example, if you are thinking about clicking a button, say why you think you need to click that button and what you expect to happen once you click it. To help you understand what I have in mind, let me give you a quick demonstration…" Some test users find thinking out loud difficult to do. If the test user starts staring quietly at the screen for a while, just ask "So, what are you thinking now?" to get the test user going again.
The only time I would not recommend asking the test user to think out loud is when it undermines what you are trying to test. For example, if you're trying to test how efficient an order entry system is by timing how long it takes to perform the task several times, thinking out loud would make any time measurements less accurate since it forces the user to work at the speed at which he can talk or find the words to describe what he is thinking.
Another important instruction you need to give is that you will only be watching the test user perform the tasks and will not be able to answer any questions about performing those tasks. You should explain that it is important to make the test as realistic as possible and real users do not have expert users standing by to answer questions. The goal of the test is to determine the user's ability to use the software, and this goal will be undermined if you tell the user how to do it. However, since you want the test user to think out loud, explain that you want the test user to ask any question she may have—you just won't be able to answer them. You should also tell the test user that you would be happy to answer any questions once the test has completed.
TIP
Avoid giving the test user assistance during the test. Encourage the test user to ask any questions—you just won't be able to answer them.
Despite these instructions, the test user may get stuck while performing a task. Do not explain this to the test user, but in this case you'll have to give the test user some help to keep the test going. If the test user is really stuck, give hints at first and more helpful information only as needed. You can answer questions the test user asks with counterquestions. For example, if the test user asks, "What happens if I click on this?" you can answer, "What do you think will happen?" Resist the temptation to give help at the first sign of struggle. Give the test user sufficient time to solve the problem before you start giving hints.
Now you are ready to roll. Ask the test user if he has any questions before you start. Perform the tests in their order of importance in case you run out of time, but make sure that the first task is fairly easy so that the test user will gain confidence. For each task, describe the task and its goal, but be careful not to suggest how the task should be accomplished. Then have the test user perform the task. Keep your eyes and ears open and your mouth shut. Never laugh at a test user's mistakes. Take good notes. At the very least, keep track of the specific user interface elements that the test user is having trouble with and what the user was thinking at the time.
TIP
During the test, keep your eyes and ears open and your mouth shut.
After the testing is complete, you should wrap up the test with an appropriate selection of the following steps:
Once a round of testing is done, you need to interpret the results since you can't necessarily take them at face value. Look for trends. If nobody has a problem performing a task, that of course is a good sign. One test user having a problem with a feature isn't necessarily significant, but multiple test users having the same problem is a clear sign of a usability problem.
When asking test users for their opinion, it's important to understand that most people want to be nice. Even if test users absolutely hate the program, to be diplomatic they often say something like, "Well, so far so good, but it clearly needs more work." Translation: "Yuck! We hate it! Better try again." I have seen this happen. Don't expect test users to list all the problems they have with a program. Some users might, but many users won't. Users' natural preference to emphasize the positive can be counterproductive in the feedback process.
TIP
Users want to be nice, so you can't always take what they say literally.
You also need to carefully interpret the test users' suggestions. If a test user's suggestions make perfect sense, you know what to do. However, if the suggestions don't make sense, try to understand why. For example, users are much better at determining problems than designing solutions, but they often phrase problems they find in terms of a suggested solution. I call this phenomenon QA Gefahren, which I discuss in detail in Chapter 36.
Interpretation of the results is necessary, but I am not suggesting that you don't take the results seriously. If you think a task is easy to do and users find it difficult, they are right and you are wrong. Don't dismiss this feedback or try to defend the feature—just fix it. The computer industry has a bad reputation for quickly dismissing feedback. For example, suppose a nuclear reactor has a meltdown because an operator accidentally pushed the wrong button. Further suppose that the reason the operator pushed the wrong button is that all the buttons on the power plant's control panel look the same. Ordinarily, one would assume that a mistake like this would be a clear sign of a design problem—the ultimate user test. In fact, such problems are often readily dismissed as operator errors or training problems, not design problems. Peter Neumanns' Computer-Related Risks is full of examples of design problems leading to "operator errors." Any mistake a test user makes is an indication that the user interface can be improved. Blaming test users for incorrectly using a poor interface prevents you from making this realization.
If the testing process revealed significant usability problems, you need to fix the problems and do another round of testing. Keep going until you get it right.