Section 33.4. A Two-Part User Study

33.4. A Two-Part User Study

We attempted to devise a test to determine how easy it is for users to misconfigure their system. After several trials, we discovered that our attempts to devise a realistic scenario involving sharing and specifying file locations became contrived and only confused our test participants. Nor could we simulate the length of time over which settings are most likely changed and then forgotten during real use. In addition, we could not speculate on all of the various reasons users would want to change their default settings, although we knew from our data that they were indeed modifying the settings and were not aware of it.

From anecdotal evidence, we know that KaZaA users often work in shared computer settings, so it is quite possible for one user to change all the settings and another to know nothing about it. We were able to contact and interview four online users who had accidentally shared their inboxes and to talk to them about their configurations. These users indicated that they had either changed the settings to make files easier to find (1 of 4) or that they shared a computer with other people who may have changed the settings without their knowledge (3 of 4). For this reason, and given the difficulties with any other realistic trial described earlier, we decided to make a very simple test to simulate this scenario. We would share all files on the hard drive (as if we were a user sharing the same machine) and see how easy it was for participants to determine what, if anything, was being shared.

Our study consisted of 12 participants: 10 of these had used file sharing applications (such as Morpheus, Gnutella, KaZaA, and Napster) before, and two had not. All the users spent more than 10 hours a week on their computers.

33.4.1. Parts of the Study

The two parts of the study are described in the following sections.

33.4.1.1 KaZaA sharing comprehension questions

We were interested in the participants' conceptions of the types of files that peer-to-peer file sharing applications can share, as well as whether they were able to perform the specified task. We asked the participants to indicate what types of files could be shared over peer-to-peer networks.

33.4.1.2 Current sharing settings discovery task

The participants were initially presented with the KaZaA interface, which had been preconfigured in the following manner:

• File sharing was enabled (thus My Shared Folder was being shared).

• The download directory was set to C:\, effectively sharing all files on the hard drive as well.

• KaZaA indexed all of C:\, and displayed the file information into the My Media folder.

In order to prevent others from downloading our files, we set up KaZaA behind a firewall and blocked incoming requests to download files. This prevented others from actually accessing our files, but still allowed KaZaA to index all the files and provide them for sharing.

Participants were given a short tutorial on file sharing and the concept of a shared folder. They were then asked to discover which files were currently being shared, if any, on a KaZaA media desktop running KaZaA version 1.7.1. All participants were given the same setup and were told to take as much time as they needed. They were allowed to use only the KaZaA interface, and at the end of their search were asked to provide a clear answer of whether they thought files were being shared, and if so, which folders they were in.

If participants correctly determined that files were being shared, we asked them to stop sharing them, and share only My Shared Folder.

33.4.2. Results

The following sections summarize the results of the two parts of the study.

33.4.2.1 KaZaA sharing comprehension questions

Only 2 of the 12 participants indicated correctly that all files could be shared. One more participant indicated correctly that it was also possible to share office documents, source code files, and email folders. The nine remaining participants believed that only multimedia files such as music, video, and pictures could be shared. After completing the task, some participants were very surprised to learn that all files could be shared with others, and some couldn't understand why. One participant exclaimed, "You mean it shares all files?" and expressed concern about why it would be able to share anything other than multimedia files. These results from our study bolster those from our cognitive walkthrough to demonstrate that KaZaA is in clear violation of the first guideline listed under "Usability Guidelines": participants should be made clearly aware of what files are being offered for others to download.

33.4.2.2 Current sharing settings discovery task

Only 2 of the 12 participants were able to determine correctly which files and folders were currently being shared. Of those 2, both were able to turn off sharing completely using the Stop Sharing feature (Figure 33-3), but were not able to determine how to stop sharing a single given folder. Of the remaining participants:

• 5 of 12 determined incorrectly that My Shared Folder was the only folder being shared, based on the information they saw from the Folder Select feature (Figure 33-7), which, as we found in our cognitive walkthrough, does not update itself based on current sharing status.

• 2 of 12 used the Find Files interface to search for folders they were sharing. When everything showed up unchecked (Figure 33-6), the participants concluded incorrectly that nothing was being shared.

• 2 of 12 browsed Help and used it to determine incorrectly that the only folder they could share was My Shared Folder.

• 1 of 12 was unable to determine what folder was being shared after going through every menu item in the application and the Help. The participant said that the files in My Shared Folder were probably being shared but admitted that he couldn't determine what, if anything, was in that folder.

Many participants found the initial interface difficult to navigate. Several participants traversed the web interface to look for help in determining what KaZaA shared, and how. In the Help section, some participants tried to use the Search function, assuming incorrectly that it searched Help and not the KaZaA network. Of the participants who were able to make it to the menus in the toolbar at the top of the application, only one was able to make the connection between the download folder (Figure 33-4) and My Shared Folder described in the help and shown on the Folder Select feature (Figure 33-7). Participants had difficulty finding the main menus in the top toolbar, and had difficulty understanding the labeling on the menu items, which made it hard for them to traverse the menu hierarchies to determine where they could control file sharing. One participant later described the experience as a "buckshot approach" to find out what was where. The participant mentioned that "he had no clue" where to look for shared folders, and resorted to looking through every menu item for something that made sense.

There was considerable confusion about My Media. Less than half of the participants thought that items in My Media were being shared with others, the rest either thought it held an archive of all media on the machine for personal use, or assumed it contained some shared and some unshared items. Only three participants could determine which items were shared and which were not by looking at the file icons, but all were unsure of which folders in My Media contained shared items and which contained items not being shared without browsing each individual folder.

33.4.3. Suggested Design Improvements

Based on what we found in the surveys, participant studies, and cognitive walkthrough, we have several suggestions that may help improve the current interface:

• Prohibit or curtail sharing of files that are not multimedia files. For example, nonmultimedia files could be shared using a more safety conscious, or advanced participant interface. As most participants in our study were unaware of the fact that they could share files other than multimedia, this would realign users' expectations with the current reality. We feel that the current interface is weighted too heavily in favor of sharing files. Our usability studies suggest that improvements can be made to create a balance between sharing files easily (to encourage KaZaA uptake by making sure there is a wealth of content available) and protecting and preserving users' privacy.

• Provide better "feedforward" about the consequences of recursion in sharing of folders and subfolders.

• Provide a more rigorous interface that explicitly supports users' efforts to make exceptions to recursively shared folders, or otherwise warns users that they have not checked subfolders in a folder that is being selected for sharing.