User Testing Goals | Developing User Interfaces for Microsoft Windows

The key to successful user testing is having practical, realistic goals, so the first step is to understand exactly what you want to accomplish. You also need to decide how much time and money you want to spend. Let's look at some practical and impractical user testing goals.

Practical User Testing Goals

For a given task, you can perform user testing to determine whether your program has the following attributes (as outlined by Nielsen):

Learnability Can the user figure out how to perform the task without assistance? Can the user perform the task at all? If so, did the user find it difficult or easy? Does the user's conceptual model of how the program works correspond to the way it actually works?

Efficiency Can the user perform the task in an acceptable amount of time? This is often referred to as time to task. The standard user testing technique of thinking out loud, described later, affects the time it takes to perform a task, so relative times (that is, comparing the efficiency of two different interfaces) are more likely to be meaningful than absolute times (that is, the time it takes to do a task using a single interface).

Few errors Can the user perform the task without making an unacceptable number of mistakes? How error-prone is the task? How serious are the errors?

Memorability Can the user remember how to perform a task after a period of time?

Satisfaction Did the user enjoy using the program? Did the user find using it a pleasant experience?

You can use these attributes to evaluate a single user interface design or you can compare two different designs by testing for any of these attributes and comparing the results.

Impractical User Testing Goals

I consider the following to be impractical user testing goals:

To find design problems not related to usability Since user testing is focused on performing tasks, it is unlikely that users will find design problems that are not task-related. A mediocre yet usable design will perform well in user testing.

To find bugs Since user testing is focused on performing tasks, it is unlikely that users will identify implementation problems. User testing is to find usability problems. It is not a substitute for QA.

To test usability for advanced users Since user testing is usually focused on beginning users, it's difficult to predict how advanced users will respond to the program and to test advanced features that require a significant amount of experience.

To discover repetitive tasks Since user testing focuses on having a user perform a task once or maybe twice to check memorability, it is difficult to determine whether a task is unnecessarily repetitive.

To obtain complete coverage Unless your program is fairly simple, it's unlikely that you can test all its features. You have to select specific tasks for the user to perform, and user testing is unlikely to find problems with features that are not associated with those selected tasks. For example, you cannot find problems with your setup program during user testing unless you explicitly make it part of your test plan.

To determine program usefulness Since you are the one selecting the tasks in the test and not the user, it's unlikely that the user would be able to determine the overall usefulness of the program by performing those tasks.

To determine marketability You cannot determine whether the user would buy the program or whether the program will be competitive.

Given all of this, what should your goals be? Of course this depends upon the type of information you want to obtain. Although you can design tasks to measure all of these attributes, in general I think it makes more sense to focus on learnability. Perform user testing to determine whether users can figure out how to use your program. If users are able to learn how to use the program quickly, chances are they will also find it easy to remember and make few mistakes. They're also likely to enjoy using the program. However, efficiency is a totally separate issue. A task that is easy to learn isn't necessarily efficient. For example, a task that is performed using a wizard is likely to be easy to learn but inefficient, whereas a task performed using keyboard shortcuts isn't learnable but is efficient. Consequently, tasks that are performed often should also be tested for efficiency. Such efficiency tests might require experienced users, since it is unlikely that a first-time user is going to become a proficient user during the test.

TIP
Focus user testing on learnability. If users find the program easy to learn, chances are they will find it efficient, not error-prone, and enjoyable as well.

You also need to select the specific tasks to test. Tasks that are performed often, that are essential to the basic functionality of the program, or that involve features that have unusual or complex user interfaces are obvious candidates. Tasks that are rarely performed or that involve features that have user interfaces similar to other successful programs are less likely to result in serious usability problems.