Section 25.4. Deployment, Adoption, and Evaluation

25.4. Deployment, Adoption, and Evaluation

So far, we have discussed how end-user privacy management systems can employ social processes to help users more effectively manage their privacy. In this section, we address how to deploy, foster adoption of, and evaluate such systems.

25.4.1. Deployment and Adoption

Successfully deploying and fostering adoption of a privacy management system that employs social processes is difficult. Such systems are groupware: software systems that support groups of people rather than individuals or whole organizations. A groupware deployment is deemed successful when enough individuals adopt the system so as to sustain its use. Several known obstacles prevent successful deployment and adoption of groupware.^[23] For privacy management systems that are also groupware, perhaps the two most significant obstacles are obtaining critical mass and mitigating the disparity between work and benefit for users.

^[23] Jonathan Grudin, "Groupware and Social Dynamics: Eight Challenges for Developers," Communications of the ACM 37:1 (1994), 92105.

A groupware system must obtain a sizable number of usersa critical massif it is to succeed. Without a critical mass of users, the system often cannot provide enough benefits to those that use the system, and eventually use of the system stops. Critical mass differs among various groupware systems. Moreover, for some groupware, it is never advantageous for any single user to use it, and thus adoption must be considered from a group perspective rather than an individual perspective.

A groupware system will fail if it requires that some or all users expend significant effort contributing to the system but obtain little benefit from it. It is important for a groupware system to balance the work required and the benefits received for all users; this is especially true for knowledgeable users who, given sufficient motivations or benefits, contribute more to the system than others.

25.4.1.1 Deployment and adoption in Acumen

Acumen's critical mass is highly correlated with the degree of coverage that its data can obtain. Acumen is most effective when it can provide data about many of the potential decisions a user may face; in other words, Acumen is most effective when it can provide user cookie management data for many different web sites. In a deployment of Acumen, we found that Acumen achieves significant coverage over the most popular web sites with as few as nine users. We discuss Acumen's coverage in more detail in this section.

In general, the critical mass for a privacy management system that employs social processes will be the number of users necessary to obtain sufficient coverage such that users find the system useful. Sufficient coverage is difficult to predict or offer guidelines for because it is dependent on the privacy management domain and on the system's users.

Acumen attempts to avoid disparities in users' efforts and benefits by using implicit data and by automatically extracting mavens' data. While it is appealing to encourage cookie management experts to contribute explicit knowledge to Acumen about how and why they are blocking particular cookies, this would increase the amount of effort that experts would have to expend without any additional benefits to them. Ultimately, experts would be less likely to participate in Acumen under these circumstances.

25.4.2. User Needs Evaluation

Evaluating a privacy management system that employs social processes can be difficult. Often, multiple sets of criteria are important when performing an evaluation.

One critical set of criteria concerns how well the system meets user needs and supports user practices. Assuming that user-centered design practices (see Chapter 2) are followed, users' privacy management needs should be clear; these needs will often fall into one or more of the three general user activities discussed earlier. The goal of evaluating a privacy management system that employs social processes, then, is to determine whether the processes supported by the system help users address their privacy management needs effectively.

Standard evaluation instruments (see Chapter 3) can be used to perform such an evaluation, but some caveats must be recognized and accounted for. The main caveat is that it is very difficult to evaluate groupware in a laboratory setting because group culture and dynamics will play a role in how the system is used. Evaluation, then, often benefits from an authentic deployment of the system to its intended users for an extended period of time. Ideally, the system should not be evaluated until users have adopted it and developed norms surrounding its use.

25.4.2.1 User needs evaluation in Acumen

We deployed Acumen to nine users for six weeks in order to perform a preliminary evaluation of the system. Users employed Acumen on a voluntary basis. We asked users to employ Acumen in the context of their normal browsing activities; we did not ask them to be more proactive in managing cookies than they otherwise would be, although the presence of Acumen's toolbar likely encouraged them to manage cookies more than they would have otherwise.

After the deployment, we obtained evaluation data using informal interviews with users, logging data, and data from Acumen's database. We used informal interviews to qualitatively assess whether and how well Acumen raised users' awareness of cookies, what effect Acumen's data had on users' decisions to block or allow cookies, and how useful mavens' data was to users. We used logging data and data from Acumen's database to obtain some objective metrics regarding system usage, such as how often users explicitly blocked cookies as compared to using rules to block cookies.

Our findings from this deployment are promising. Acumen does raise users' awareness of cookie management and helps users manage their cookies in an informed way. Because privacy is highly subjective, it is difficult to assess whether users allowed "good" cookies and blocked "bad" cookies, but our data indicates a degree of consensus among users for some cookies.

For example, advertisers' cookies (e.g., atdmt.com and hitbox.com) were often blocked by a significant number of users; web sites that use cookies to provide personalized services (e.g., yahoo.com and amazon.com) were often allowed. There were, however, some sites for which users disagreed about whether to block or allow cookies (e.g., msn.com and google.com); often such sites were well known and trusted but offered users little or no tangible benefits for the use of cookies. We speculate that users consider two factors when deciding whether to allow or block a site's cookies: first, trust in the site; and second, the benefit/cost ratio associated with the site's use of cookies. We report our complete findings from this deployment elsewhere.^[24]

^[24] Jeremy Goecks and Elizabeth D. Mynatt, "Supporting Privacy Management via Community Experience and Expertise," Proceedings of the Second Communities and Technologies Conference (Milano 2005).

25.4.3. Technological Evaluation

Systems that support social processes by making users' activities visible to each other have some rather unique technological requirements. Hence, it is often useful to evaluate such systems from a technological standpoint.

We previously discussed the relationship between achieving critical mass and data coverage in a privacy management system that utilizes social processes. Data coverage is both a social and a technological issue; coverage reflects not only which activities users perform, but also the ability of the system to capture and record data.

A purely technological issue for a privacy management system is its response time. Privacy management systems are most effective when they respond quickly to user input. Real-time interaction enables users to experiment with privacy management; as discussed earlier, experimentation is valuable for learning. Achieving real-time interaction, however, can be difficult to do in systems that are driven by large numbers of users and by the data that they generate. Replicating system components and databases and using cached but perhaps slightly inaccurate data may be acceptable if the system appears to respond in real time to users. (In order to respond in real time using cached data, a system would update the cached data based on a user's action and show the updated, cached data to the user.)

25.4.3.1 Technological evaluation in Acumen

Data coverage appears to be a challenging issue for Acumen. For the purposes of this discussion, we assume that Acumen provides data coverage for a web site if two or more users have visited the site. Thus, any user who visits the site can observe how at least one other person manages the site's cookies.

There are millions of web sites, and Acumen will not contain data for every siteor likely even most sites. However, Acumen's data is tied to users' browsing activities; users will manage only cookies that they encounter while browsing. Hence, Acumen's data coverage mirrors the coverage obtained by users' browsing activities. Traffic among web sites on the Internet has been shown to obey a power law distribution.^[25] A basic power law distribution for web traffic says that the nth most popular web site receives about 1/n as much traffic as the most popular web site. It follows that traffic to the most popular sites is a very, very large proportion of total traffic.

^[25] Bernardo Huberman, The Laws of the Web: Patterns in the Ecology of Information (Cambridge, MA: MIT Press, 2001).

Acumen, then, should contain data for the most popular web sites and be significantly less likely to have data for less popular sites. Data obtained from our deployment of Acumen confirmed this hypothesis:

• Among the most popular 20% of web sites, nearly half are covered by Acumen's data.

• Among the most popular 40% of web sites, about two in five are covered.

• Among the most popular 60% of web sites, one-third are covered.

Power law distributions are quite common in human activity, and they can mitigate issues of data coverage in other systems that employ social data.

Acumen achieves real-time interaction with users by utilizing a modular, replication-based architecture that distributes user interaction and data processing across many components (see the sidebar, "Acumen Architecture"). Acumen utilizes multiple instances of the components that users interact with to ensure that the system responds in real time. Each component has a very short-term cache that decreases response time as well; data is cached for 13 seconds.