Section 22.4. Privacy Bird Evaluation

22.4. Privacy Bird Evaluation

We performed two studies to evaluate the usefulness and usability of Privacy Bird, investigating how it is used in a controlled laboratory setting as well as how it is used in practice. We conducted a laboratory study that allowed us to make detailed firsthand observations of how first-time users interacted with the Privacy Bird software and to compare Privacy Bird with another P3P user agent. In addition, we were able to observe users performing the same tasks with and without the benefit of a P3P user agent and thus evaluate the effectiveness of the user agent. We also conducted a user survey to gather information about how Privacy Bird is used in practice. This survey provided us with self-reported data from individuals who had been using the software for several months in their own homes or offices.

22.4.1. User Survey

We received informal feedback on our first beta release of Privacy Bird from demo audiences and from some of the approximately 30,000 users who downloaded it. Email from our users focused on requests for new features and ports to other platforms, and stability and compatibility problems. In order to get additional feedback and gain a better understanding of how people were actually using Privacy Bird, we conducted a survey of Privacy Bird users in August 2002.^[28] We sent email invitations to complete a 35-question online survey to 2,000 of the email addresses provided by individuals who had downloaded Privacy Bird during the first six months of our beta trial and had given their permission to be contacted for user studies. We received 309 completed surveys.

^[28] Lorrie Faith Cranor, Manjula Arjula, and Praveen Guduru, "Use of a P3P User Agent by Early Adopters," Proceeding of the ACM Workshop on Privacy in the Electronic Society (ACM Press, 2002), 110.

We asked respondents to evaluate how easy or difficult it was to use several aspects of Privacy Bird. Because our results indicated that users had the most difficulty in understanding the Policy Summary, we focused most of our attention on that aspect of Privacy Bird for the beta 1.2 release.

A frequent criticism respondents had of Privacy Bird was that a yellow bird appeared at most web sites (because most web sites are not yet P3P enabled^[29]). The survey indicated that respondents would find Privacy Bird considerably more useful if most web sites were P3P enabled, and if Privacy Bird were capable of blocking cookies at web sites where the red bird was displayed.

^[29] In August 2002, Ernst & Young reported that 24% of the top 100 domains and 16% of the top 500 domains visited by U.S. Internet users had been P3P enabled (see http://www.ey.com/global/download.nsf/US/P3P_Dashboard_-_ _August_2002/$file/P3PDashboardAugust2002.pdf).

We asked users whether they had learned anything about web site privacy policies as they used Privacy Bird that caused them to change their online behavior. A total of 88% indicated that their use of Privacy Bird had resulted in some change in behavior: about 37% of respondents reported that they fill out fewer forms online; 37% reported taking advantage of opt-out opportunities; 29% reported that they stopped visiting some web sites; and 18% reported comparing privacy policies at similar sites and trying to frequent the sites with the better privacy policies. While the fact that these are responses from self-selected survey respondents is probably a factor, these results do suggest that P3P has the potential to influence user behavior.

As a result of this study, we made several changes to the Privacy Bird interface before releasing the beta 1.2 version.

22.4.2. Laboratory Study

We conducted a laboratory study involving 12 Microsoft Internet Explorer users who had never used Privacy Bird or the P3P features in IE6.^[30] Subjects were given a brief tutorial on Privacy Bird beta 1.2 and the IE6 P3P features and were then asked to use these tools to answer several questions about a web site's privacy policy. As a control, they were also asked to read an English-language privacy policy at a different web site and answer the same questions. Subjects filled out pre-test and post-test questionnaires and discussed their experience with a moderator.

^[30] Lorrie Faith Cranor, Praveen Guduru, and Manjula Arjula, "User Interfaces for Privacy Agents," ACM Transactions on Computer Human Interactions (TOCHI) (2006, in press).

Subjects were asked to respond to questions and follow instructions provided by a web-based interface on a personal computer running Windows NT and IE6. This interface allowed us to record the subjects' responses and to collect information automatically about how long it took the subjects to perform each task.

Each subject was asked to perform a set of tasks using Privacy Bird and IE6, and by reading a site's English-language privacy policy. Subjects were randomly assigned an order in which to complete these three sets of tasks. The tasks involved visiting a specified well-known commercial web site and answering four questions frequently asked about web site privacy policies. The four questions required subjects to determine:

• Whether the site might send a visitor unsolicited email

• Whether the site might share a visitor's email address with another company that might send the visitor unsolicited email

• Whether the site uses cookies

• What steps a visitor could take to exercise opt-out or unsubscribe options

Post-test questionnaires asked subjects to rate several aspects of the ease of use of each user agent and the likelihood that they would use it in the future or recommend it to a friend.

Subjects found using either P3P user agent preferable to reading web site privacy policies; however, they preferred Privacy Bird to IE6. Many subjects remarked that they liked the structured nature of the Privacy Bird policy summary and found the bulleted items easy to read and understand. They liked the fact that Privacy Bird presents information in a consistent format. They also remarked that although the IE6 policy summary uses a standard format, they found it to be far too verbose, which made it difficult to quickly scroll through it to find particular information. In fact, we observed that some subjects attempted to use the browser's search feature to find information in both the English-language privacy policies and in the IE6 policy summary. When searching English-language privacy policies, they usually had to try several terms until they figured out what terminology a particular web site was using to describe a given data practice, and sometimes this strategy proved ultimately unsuccessful. They were unable to search the IE6 policy summary as no search tool is provided. Furthermore, the IE6 policy summary does not include important information needed to answer some of the questions we posed to our subjects.

Our results suggest that individuals who are looking for a specific piece of information in a privacy policy will likely find that information faster using Privacy Bird than using IE6 or reading the policy. The Privacy Bird policy summary could be further improved so that it highlights information that users are most likely to seek and provides a summary of this information across all the statements in a P3P policy. The policy summary might also be customized based on the privacy preferences specified by each user. Observations of users making privacy-related decisions while browsing the Internet in their own home or work environments would help inform policy summary refinements.