Section 12.3. Usability Testing of Security Devices

12.3. Usability Testing of Security Devices

Security devices, like other security technologies, are deployed in a physical and social context to allow users to complete their tasks. Thus, it is essential to perform usability studies on the system as a wholein other words, taking into consideration the specific contexts, users, and tasks users must perform.^[10] In the testing described in this chapter, we have tried to capture many of the important usability attributes that will be applicable generally. Do keep in mind, however, that while we are striving for general attributes, you should review your own actual environment to identify additional attributes that might affect your usability, or drop one or more of those we have identified.

^[10] Sasse, Brostoff, and Weirich.

12.3.1. Setting Up the Test

Our attributes definition draws on prior research^{[11], [12]} and the ISO 9126 standard.^[13] The ISO 9126 software quality model defines six independent software quality characteristics, each of which is broken down into a set of subcharacteristics. However, usability studies of security devices deal not only with software but also with systems. Therefore, while the ISO standard is an important reference point, we have had to extend and adapt it to accommodate our specific aspects of interest.

^[11] B. Pinkas and T. Sander, "Securing Passwords Against Dictionary Attacks," Proceedings of the 9th ACM Conference on Computer and Communications Security (ACM Press, 2002), 161170.

^[12] A. Whitten and J. D. Tygar, "Why Johnny Can't Encrypt: A Usability Case Study of PGP 5.0", Proceedings of the 8th USENIX Security Symposium (Aug. 1999). See also Chapter 34, this volume.

^[13] ISO, International Standard ISO/IEC 9126. Information technologySoftware product evaluationQuality characteristics and guidelines for their use, International Organization for Standardization, International Electrotechnical Commission, Geneva (1991).

The usability attributes we adopted are an extension of the following three ISO 9126 usability subcharacteristics:

• Learnability. How difficult/easy is it for users to learn how to use the security devices?

• Operability. How difficult is it for users and their organizations to carry out the assigned tasks properly while using the security devices?

• Attractiveness. To what extent are the security devices attractive to users?

Operability is a very broad attribute. Therefore, we have broken it down into the following subattributes:^{[14], [15]}

^[14] Pinkas and Sander.

^[15] Whitten and Tygar.

• Mobility. Organizations' procedures and structures are increasingly encompassing remote and mobile employees. Are applications that require security devices easily operable from different computers?

• Installability. In order to operate security devices properly, users must be able to install them. Can users do so without too much difficulty?

• User friendliness. Can users operate the devices easily? Are the devices prone to user error?

• Low operating costs. Do the security devices require customer support calls? Is too much user time spent merely trying to interact with the devices?

• Security interaction. Do the security devices provide enhanced security without creating the potential for dangerous error? More specifically, what influence, if any, does the devices' usability have on security? Do usability problems lead to security failures?

One controversial topic is how security devices that require a reader should be evaluated using this framework. A company that is promoting these devices might want an evaluation to consider only the device itself, and assume that the readers are ubiquitous. We take an opposing view: given today's computer hardwarethat is, the system's physical contextthese issues will be raised unless the reader is already in place. This situation can create both usability and security issues if the user is forced to carry around extra hardware and software in order to operate the device.

Here, we must make some additional comments about the adaptation of the ISO model of software quality to the specific type of systems in our study. The first comment concerns the hierarchical relationship between the attributes. In our approach, each subattribute of operability has the same relevance level of learnability and attractiveness; the subattributes are simply classified under the common name of operability. The second comment is related to the security interaction; somewhat arbitrarily, we have placed it under the head operability. In fact, the ISO standard does not model this type of attribute, because the attribute does not describe a property in a hierarchy, but an interaction between properties (usability and security).

Our experimental approach is driven by the usability attributes defined in this section. In other words, each attribute will be evaluated through a set of metrics detailed in the experiments.

12.3.2. Related Work

Little research has been published on the usability of security devices. One recent study performed by Wu et al. compares the usability of mobile-phone-based authentication interfaces.^[16] Herzberg,^[17] and Mallat et al.^[18] describe financial services that can be enabled through mobile phones, but their focus on usability is minimal. Fulcher discusses the deployment of a cryptographic USB token in an "eHealth" project, but neither analyzes its usability nor discusses usability tradeoffs of other security devices that might be used in the same context.^[19] The usability characteristics of other types of authentication mechanisms are discussed in Part II of this book.

^[16] M. Wu, S. Garfinkel, and R. Miller, "Secure Web Authentication with Mobile Phones" [cited 11/2004]; http://www.mit.edu/~minwu/.

^[17] A. Herzberg, "Payments and Banking with Mobile Personal Devices," Communications of the ACM 46:5 (May 2003).

^[18] N. Mallat, M. Rossi, and V. K. Tuunainen, "Mobile Banking Services," Communications of the ACM 47:5 (May 2004).

^[19] J. Fulcher, "The Use of Smart Devices in eHealth," Proceedings of the 1st International Symposium on Information and Communication Technologies, Trinity College Dublin (2003), 2732.

A recent study by Scholtz and Consolvo^[20] proposes a framework for conducting a usability study of ubiquitous computing applications. The goals of their frameworkthat is, to allow researchers to learn from each other's results and to create effective evaluation techniques and design guidelinescould apply equally to the evaluation of security devices. The approach we propose in this chapter may be considered a first step in defining a more complete framework similar to the one proposed by these researchers.

^[20] J. Scholtz, S. Consolvo, "Towards a Discipline for Evaluating Ubiquitous Computing Application," National Institute of Standards and Technology and Intel Research [cited 01/2004]; http://www.intel-research.net/Publications/Seattle/022520041200_232.pdf.

12.3.3. Usability Testing Methodology

The following steps summarize the methodology we adopted in our usability testing:

1. Purpose and scope definition. We define the aims of the test (e.g., comparing the usability of two types of devices) and set the test's limits.

2. Context and roles definition. We define the context for the experimental scenario, including the simulated environment, user roles, tasks they need to achieve, and so on. Each role must be specified clearly, including the possible actions of a supervisor.

3. User selection. We define the selection criteria of users based on the selected context and aims of the test. We must select a user sample wide enough to assure statistical significance.

4. Task definition. We define the set of tasks to be executed by each user (sequence of steps, input data, output data).

5. Measurement apparatus design. We choose a set of metrics and specify their relationships with the usability attributes. For each metric, we define name, description, scale, and procedure to collect the raw data and compute the measurement.

6. Execution and data collection. We execute the test and collect the data.

7. Processing for statistical significance. We process the data to ensure its statistical significance.

8. Computation of the quality attributes scores. We compare the quality profile of each device. In the ISO 9126 quality model, product usability is linked to a set of attributes, and each attribute is evaluated through a set of metrics. Note that this procedure has to consider various types of measures, including both quantitative and qualitative ones.

9. Results interpretation and explanation. We interpret our set of results and suggest possible causal explanations that will be useful for generating design recommendations.