Section 34.3. Evaluation Methods

34.3. Evaluation Methods

We chose to evaluate PGP's usability through two methods: an informal cognitive walkthrough^[12] in which we reviewed PGP's user interface directly and noted aspects of its design that failed to meet the usability standard described in the preceding section, and a user test^[13] performed in a laboratory with test participants selected to be reasonably representative of the general population of email users. The strengths and weaknesses inherent in each of the two methods made them useful in quite different ways, and it was more realistic for us to view them as complementary evaluation strategies^[14] than to attempt to use the laboratory test to directly verify the points raised by the cognitive walkthrough.

^[12] Wharton.

^[13] Rubin.

^[14] B. E. John, and M. M. Mashyna, "Evaluating a Multimedia Authoring Tool with Cognitive Walkthrough and Think-Aloud User Studies," Journal of the American Society of Information Science 48:9, 1997.

Cognitive walkthrough is a usability evaluation technique modeled after the software engineering practice of code walkthroughs. To perform a cognitive walkthrough, the evaluators step through the use of the software as if they were novice users, attempting to mentally simulate what they think the novices' understanding of the software would be at each point, and looking for probable errors and areas of confusion. As an evaluation tool, cognitive walkthrough tends to focus on the learnability of the user interface (as opposed to, say, the efficiency), and as such it is an appropriate tool for evaluating the usability of security.

Although our analysis is most accurately described as a cognitive walkthough, it also incorporated aspects of another technique, heuristic evaluation.^[15] In this technique, the user interface is evaluated against a specific list of high-priority usability principles; our list of principles is comprised by our definition of usability for security (in the section "Defining Usability for Security") and its restatement specifically for PGP (in the section "A Usability Standard for PGP"). Heuristic evaluation is ideally performed by people who are "double experts," highly familiar both with the application domain and with usability techniques and requirements (including an understanding of the skills, mindset, and background of the people who are expected to use the software). Our evaluation draws on our experience as security researchers and on additional background in training and tutoring novice computer users, as well as in theater, anthropology, and psychology.

^[15] Jakob Nielsen, "Heuristic Evaluation," Usability Inspection Methods (New York: John Wiley & Sons, Inc., 1994).

Some of the same properties that make the design of usable security a difficult and specialized problem also make testing the usability of security a challenging task. To conduct a user test, we must ask the participants to use the software to perform some task that will include the use of the security. If, however, we prompt them to perform a security task directly, when in real life they might have had no awareness of that task, then we have failed to test whether the software is designed well enough to give them that awareness when they need it. Furthermore, to test whether they are able to figure out how to use the security when they want it, we must make sure that the test scenario gives them some secret that they consider worth protecting, comparable to the value we expect them to place on their own secrets in the real world. Designing tests that take these requirements adequately into account is something that must be done carefully, and with the exception of some work on testing the effectiveness of warning labels,^[16] we have found little existing material on user testing that addresses similar concerns.

^[16] M. S. Wogalter and S. L. Young, "Enhancing Warning Compliance Through Alternative Product Label Designs," Applied Ergonomics 25 (1994), 357.