The Day After

What did we achieve? Did the product get better because of the test tour? Our concrete data consisted of a pile of display pictures with a lot of hand-drawn corrections and scribbled additions, video and audiotapes, and observation notes. Once we had combed through them, we agreed that the test engendered a number of insights, provided information about the acceptability of the new Series 60 user interface style, pinpointed specific design problems, and yielded several new design and localization ideas. The results are tabulated in Figure 8.5.

Figure 8.5: The distribution of different types of results in a paper prototyping test.

Dumash and Redish^[7] have created a classification of software usability problems. Figure 8.6 lists the main problems our tests uncovered and maps them to corresponding items in Dumash and Redish's classification. Their categories capture half of the problem types that we observed during our test round in four countries. The figure also shows that most of the observed problems could be fixed to final design.

As previously mentioned, paper prototyping is a tool to catch big fish in usability testing. However, the problems listed in Figure 8.6 seem like relatively minor issues. Does this suggest that we were wrong to assume that paper protos go to the very basic questions of a user interface style? Not necessarily. An excessive number of menu items, screen layouts that do not give hints about options not immediately displayed, and wasting time with data entry are either examples of problems that bring the whole style into question or just indicate unfinished design. It's up to the research team to analyze the implications. This time it was possible to fix the problems without having to scrap the basic core interaction logic.

The tests in Rome belonged to a series of four carried out in Finland, Italy, Germany, and the United Kingdom with 38 test subjects. The results from Rome confirmed, contradicted, or were neutral with respect to other findings in the research project as a whole. Several findings recurred in all test tasks and test sessions, and most of the usability problems were observed in at least two countries. For example, a specific but not obvious menu command that was not in the prototype was requested by a user in every country. This kind of observation can be regarded as a reliable conclusion and as a severe usability problem that has to be fixed. Contradictory findings were usually users' preferences concerning the order of menu items or comments about localization issues. Some problems were seen in only one country and they most often concerned a missing note or a text that was hard to understand, requiring either language-specific fixes or the addition of clarifying information.

Figure 8.6: Observed usability problems and mapping to the Dumash-Redish classification.⁷

^[7]J. S. Dumash and J. C. Redish, A Practical Guide to Usability Testing. Portland, Oreg.: Intellect Books, 1999.