The SCAPE Method

---

Later in this chapter, we will give an example of how SCAPE was used to evaluate the awareness and privacy support of a particular application being considered for use by a retailer and supplier. Because we will give examples of developing SCAPE analysis materials that were culled from our case study, we introduce the case study scenario in the next subsection. The fine-grained awareness framework and awareness relationships that underpin SCAPE are introduced in the following subsections, ending this section with a by a step-by- step description of the SCAPE method.

Case Study Scenario

We developed a scenario based on recent events in the popular business press (e.g., (Bianco & Zellner, 2003)). The scenario centers on the partnership between a fictitious major retailer (“Wal-Store”) and an equally fictitious nationwide distributor of juice drinks (“Sea Spray”).

Joy Brown is a beverages buyer at Wal-Store, and Jenn Smith is a manufacturer’s representative from Sea Spray. Jenn works with two product managers, Rod Leeds and Sally Steele. Joy wants to get the Sea Spray juice drink products on the shelves at Wal-Store at the lowest possible price by probing Sea Spray’s manufacturing, inventorying, and distribution processes and suggesting ways to streamline these processes. Jenn would like to get as many different Sea Spray products on Wal-Store’s shelves as possible and so is willing to work with Joy— up to a point. She does not want to give Joy information about the recipes for the newest juice drinks because Wal-Store has a history of using “inside” product information to manufacture similar products under the “Sal’s Choice” in-house label, thus eating into the market share of national brands.

Jenn would like to use the same IOIS with Joy as well as Rod and Sally, her product managers for the two new juice drinks. She would like to keep the proprietary information private from Joy, however.

By highlighting the natural tension between competitors, this Wal-Store scenario includes the notion of adversarial collaboration (Cohen, Cash, & Muller, 2000): situations in which collaborators have widely divergent goals yet must work together to perform specific tasks. Other situations that involve adversarial collaboration are merger/acquisition negotiations and document sharing among opposing legal teams. We expect that, as technology becomes more widely used to facilitate communications between organizations with divergent goals, computer-based adversarial collaboration support will become increasingly more important.

Both Cohen et al.’s study and the Wal-Store scenario illustrate a need for IOISs to support detailed awareness and privacy requirements and to ensure that everything that is visible to others is revealed intentionally instead of being revealed accidentally or by default.

Awareness Framework

Before describing the SCAPE method, we first need to refine the definitions of awareness cited earlier into a more fine-grained analysis framework. Although there are many definitions of awareness (a few of which were cited above), there is no standard definition. We define awareness as follows (Drury, 2001):

Awareness - Given two participants p₁ and p₂ who are collaborating via a synchronous collaborative application, awareness is the understanding that p₁ has of the identity and activities of p₂.

To support p₁’s understanding of p₂, an application provides p₁ with information about the identity and activities of p₂ without p₁ having to request the information or p₂ having to explicitly transmit it. Awareness information is intended to emulate the kinds of nonverbal cues that people get about each other when they work face to face in the same physical environment.

The awareness information that p₁ might have access to regarding p₂ includes (but is not necessarily limited to) p₂’s presence in the shared workspace, p₂’s identity, the task that p₂ is performing, the tools and artifacts that p₂ is using, the changes p₂ is making, the area of the workspace viewable by p₂, and p₂’s focus within that viewable area. We refer to the aggregation of all of the awareness information that all of the participants may have about each other as the awareness information space.

Although the awareness literature largely assumes that the more awareness information available, the better, there are times when awareness information should be withheld. For our purposes, we define privacy as follows:

Privacy - The intentional withholding of awareness information.

Evaluation of awareness support has two underlying principles: (1) ensure that awareness information that should be provided is provided; and (2) ensure that awareness information that should not be provided is kept private. A different type of awareness-related error is associated with each of these principles (Drury & Williams, 2002):

Type 1 - Awareness violation. Awareness information that should be provided is not (a violation of the first principle).

Type 2 - Privacy violation. Awareness information that should be kept private is revealed (a violation of the second principle).

A method for evaluating awareness support must provide for the identification of awareness violations and privacy violations.

There is currently no theory per se of awareness support. We constructed a general-case default specification for the kinds of awareness information after examining eight theories and frameworks for collaborative work [see Drury & Williams (2002)]. We developed a general, application-independent awareness specification, expressed as heuristics (see Figure 1). These awareness heuristics are of two types: activity heuristics pertaining to activities only, regardless of who is performing them; and identity heuristics pertaining to participants.

Figure 1: General awareness specification, expressed as activity heuristics and identity heuristics. This specification is application-independent.

The general awareness specification has two assumptions built into it: (1) all participants using a collaborative application have the same awareness needs, and (2) participants require access to all possible awareness information about each other. The general specification may be tailored for specific applications where these assumptions do not hold.

We refer to the situation where a participant has access to all possible awareness about another participant as complete awareness:

Complete awareness - Given two participants p₁ and p₂ who are collaborating via a synchronous collaborative application, if all information regarding p₂ in the awareness information space is available to p₁, we say that p₁ has complete awareness with respect to p₂.

It is common for a participant to need (or to be permitted) only limited, rather than complete information about another participant, so that partial awareness should be provided:

Partial awareness - If some, but not all, information regarding p₂ in the awareness information space is available to p₁, we say that p₁ has partial awareness with respect to p₂.

Whether a participant p₁ needs complete or partial awareness of another participant p₂ depends on the roles played by p₁ and p₂. For example, in the application supporting the retail scenario, p₁ may be a buyer and p₂ may be a sales representative. We define a role to be a participant’s activities and responsibilities with respect to the other participants in a collaborative session.

Participants using a collaborative application can be partitioned into role-based equivalence classes (e.g., buyer, sales representative). All participants in the same role have the same awareness and privacy needs.

Awareness Relationships

Each role is related to every other role by an awareness relationship that characterizes the awareness information participants in one role may have about participants in another.

An awareness relationship may be either complete or partial, depending whether participants have complete or partial awareness of each other. It is also possible for participants in one role to have no awareness information about participants in another role, in which case we describe the awareness relationship as no awareness.

Awareness relationships are unidirectional. Given two roles r₁ and r₂, there is an awareness relationship for r₁ with respect to r₂, characterizing the information available to participants in r₁ about participants in r₂. The relationship may be complete, partial, or no awareness. Similarly, there is an awareness relationship for r₂ with respect to r₁. It also may be complete, partial, or no awareness, independent of the type of the relationship for r₁ with respect to r₂.

Figure 2 shows a matrix of awareness relationships for evaluating a hypothetical collaboration application used by buyers and sales representatives, assuming use by one participant in the buyer role, one in the sales representative role, and two in the product managers’ roles. Some roles (e.g., product managers) have awareness relationships with themselves, which characterize what participants in the same role may know about each other (what one product manager may know about another). Because there is only one buyer (and one sales representative), there is no buyer–buyer (or sales representative–sales representative) awareness relationship.

Figure 2: Awareness relationship matrix for the hypothetical application used by retailers and suppliers. (Numbers indicate the number of participants in each role.)

Steps for Performing a SCAPE Evaluation

SCAPE is based on the awareness framework and awareness relationships. It can be used to determine whether users’ awareness and privacy needs would be met by an application. In particular, SCAPE is especially useful for evaluating a synchronous collaborative application, because such systems require an up-tothe-moment awareness of fellow participants’ identities and activities. Figure 3 illustrates the difference in awareness needs for synchronous versus asynchronous applications. A “yes” under the “Sync.” column means the heuristic is applicable to synchronous applications; similarly, a “yes” in the “Async.” column implies that the heuristic pertains to an asynchronous application. A “no” in either column means the heuristic is not relevant for that type of collaborative application.

Figure 3: Awareness needs in synchronous versus asynchronous applications. (“Yes” means the heuristic is applicable; “No” means the heuristic is not applicable.)

SCAPE is an inspection method performed by evaluators, and it is based on the awareness framework described earlier in this section. It is designed to help evaluators find both awareness violations and privacy violations. SCAPE takes into account the roles that participants play and the awareness relationships between the roles.

The SCAPE method has two parts: (1) development of a detailed, application- specific specification of awareness and privacy requirements; and (2) evaluation of the application for compliance with the specification. The SCAPE Handbook (Drury, 2001) contains more detailed explanations than can be included here, as well as examples, advice, and worksheets.

There are three steps to developing an awareness specification for a collaborative application: define the awareness relationships, develop role-based awareness policies, and identify activity-based exceptions to the policies.

Step 1 - Define awareness relationships. The goal of this step is to identify roles and the high-level awareness relationships between them. Knowledge of the application domain is needed to perform this step. Roles are identified, and an awareness relationship matrix, such as the one shown in Figure 2, is created. The output of this step is the matrix.

Step 2 - Develop role-based awareness policies. The goal of this step is to develop a set of awareness policies, based on the relationships between roles. The approach is to modify the general awareness specification from Figure 1 according to the awareness relationships identified in Step 1. While awareness policies can be created for each role pair, in practice, usually only one awareness policy is needed; it indicates the superset of what participants can know about other participants. Exceptions to this policy are identified in the next step. The awareness policy for the case study is shown in Figure 4.

Figure 4: Awareness policy for the case study. (None of the awareness heuristics are deleted in this example, and no extra heuristics have been added. Notations to the worksheet are indicated by the Comic Sans MS font.)

For each awareness relationship, the evaluator begins with the general awareness specification and deletes any portions of the specification that are not applicable. (For example, if the application requires anonymity, then all identity heuristics are deleted.)

We also leave open the possibility that the evaluator may need to add to the specification. The evaluator performs this step as many times as necessary:

• Once for all of the complete awareness relationships, because they all have the same requirements

• Once for each partial awareness relationship, because they may all have different requirements

The output of this step is a set of role-based awareness policies expressed as heuristics.

Step 3 - Identify activity-based exceptions. The goal of this step is to tailor the role-based awareness policies developed in Step 2 so that they include any activity-related exceptions that are necessary to ensure privacy. All of the policies for partial awareness relationships must be tailored in this way. (There is no need to tailor the policies for complete and “no awareness” relationships.)

The evaluator begins with the awareness relationship matrix from Step 1 and the set of awareness policies from Step 2, and uses domain knowledge about tasks that participants will perform. The evaluator identifies the activities during which parts of an awareness policy should be suspended. (For example, it may be OK for a product manager to be aware of what a buyer is doing, except when the buyer is working on another purchase in which the product manager is not involved.)

The result can be seen in Figure 5, which is a portion of a privacy policy for the case study. The set of tailored awareness and privacy policies constitutes an application-specific awareness requirements specification.

Figure 5: Sample privacy policy for the case study, showing what the buyer cannot know about the sales representative’s activities.

There are three steps to evaluating an application for compliance with an awareness specification: assess the level of effort a complete evaluation would entail, develop scenarios to use during evaluation, and perform the evaluation.

Step 4 - Assess level of effort. The goal of this step is to assess the level of effort that a full-fledged evaluation would entail. A SCAPE evaluation involves a series of mini-evaluations, because each awareness relationship needs to be evaluated. The level of effort can be substantial, because the worst-case number of awareness relationships can be roughly estimated as the square of the number of roles. Thus, it is prudent to calculate the level of effort needed for a complete SCAPE evaluation, compare the result to the resources available, and select the highest-priority awareness relationships to evaluate.

The evaluator begins with the role relationship matrix from Step 1, the specification from Step 3, and an understanding of resources (time, money, etc.) available. If the evaluator expects that evaluating one awareness relationship will take e effort (measured in evaluators’ time, money, etc.), and if there are a awareness relationships, then the level of effort for a complete evaluation is proportional to e multiplied by a.

To identify the awareness relationships that are the highest priorities for evaluation, the evaluator annotates the awareness policies, indicating whether violations would have high, medium, or low impact. The output is a list of prioritized awareness relationships.

Step 5 - Develop scenarios. The goal of Step 5 is to specify activities that will cause SCAPE evaluators to perform sequences of actions that will exercise the application’s awareness support capabilities. Scenarios are a well-established technique for evaluation; Nielsen pointed out the advantages of using scenarios for heuristic evaluation when it is important to examine participants’ interactions (Nielsen, 1995). The evaluator uses the awareness specification from Step 3 and the prioritized list of awareness relationships from Step 4 to develop a master scenario and scenario worksheets to use during the evaluation. The master scenario worksheet from the case study can be seen as an example in Figure 6.

Figure 6: Example portion of master scenario worksheet developed for the case study.

Step 6 - Perform the evaluation. The goal of this step is to identify awareness violations and privacy violations. Teams of evaluators perform the scenarios developed in Step 5 and examine the application for compliance with the specification from Steps 2 and 3. The output of this step is a set of problem reports.

Before performing a SCAPE evaluation, it is critical that the evaluators understand the users’ tasks and roles and the applications’ functionality. Without a good understanding of the users, the evaluation results may not predict how well the application will meet users’ needs (that is, the evaluation may point out issues that the users would not consider problems or may miss problems entirely). To understand the users, we performed a task analysis and critical incident analysis based on structured interviews with a buyer for a large Eastern-U.S. retailer. One of the evaluators already had significant prior experience with using the collaborative application and helped train the other (supplementing training provided by the developer’s tutorial information).

An explanation of the evaluation performed for the case study can be found below, after a description of the application evaluated.

---