Usability Tests

1. Listen. As tempting as it is to immediately discuss what you're observing, make sure to listen to what people are really saying. Feel free to discuss what you're seeing, but don't forget to listen.

2. Usability tests are not statistically representative. If three out of four people say something, that doesn't mean that 75% of the population feels that way. It does mean that a number of people may feel that way, but it doesn't mean anything numerically.

3. Don't take every word as gospel. These are just the views of a couple of people. If they have good ideas, great, but trust your intuition in judging their importance, unless there's significant evidence otherwise. So if someone says, "I hate the green," that doesn't mean that you change the color (though if everyone says "I hate the green," then it's something to research further).

4. People are contradictory. Listen to how people are thinking about the topics and what criteria they use to come to conclusions, not necessarily the specific desires they voice. A person may not realize that two desires are impossible to have simultaneously, or he or she may not care. Be prepared to be occasionally bored or confused. People's actions aren't always interesting or insightful.

5. Don't expect revolutions. If you can get one or two good ideas out of each usability test, then it has served its purpose.

6. Watch for what people don't do or don't notice as much as you watch what they do and notice.

For in-room observers, add the following instructions:

7. Feel free to ask questions when the moderator gives you an explicit opportunity. Ask questions that do not imply a value judgment about the product one way or another. So instead of asking, "Is this the best-of-breed product in its class?" ask, "Are there other products that do what this one does? Do you have any opinions about any of them?"

8. Do not mention your direct involvement with the product. It's easier for people to comment about the effectiveness of a product when they don't feel that someone with a lot vested in it is in the same room.