Testing of the Abacus Prototype

Thirty-six academics, university students and local government employees who were known to be users of ESD, took part in the study. Of these, five were deemed to be experts with ESD and, although the study was concerned with casual users of ESD, the experts were included in the usability testing for the purposes of comparison with the casual users. A sample of 36 participants is comparable to those in studies described by Nielsen and Levy (1994).

The Usability Testing Procedure

Usability tests were conducted with one volunteer at a time under identical conditions. After a scripted introduction, all participants were given the same three sets of printed material:

A set of training activities to be carried out using the prototype.
A set of typical tasks to be carried out using the prototype.
An evaluation sheet.

Each participant's interactions with the prototype were video recorded.

It was necessary to familiarize the participants with the prototype prior to their carrying out a set of representative tasks. To minimize both variability and bias in the training regime, it was decided neither to instruct participants verbally nor to allow them to experiment, at length, with the prototype. Instead, all participants were given the same set of written instructions to follow in a hands-on training session with the prototype.

One of the objectives of the usability testing was to learn whether participants could apply different OLAP functions to complete tasks. For this reason, the training materials did not present participants with one or two simple strategies that could be used to complete all the tasks. Such a training approach would only allow us to determine whether participants could select the appropriate strategy for a particular type of task. Instead, the training materials only showed the participants how to select a database and how to use the various OLAP functions, not how to combine these functions to complete tasks. Thus, the development of strategies was left to the participants.

A set of 18 tasks was developed and organized as follows:

Seven tasks in Table Mode.
Seven tasks in Value Mode.
Four Mixed Tasks - tasks for which the participant could select either mode.

A typical task would be to "find the number of single people in NSW who earn over $80,000."

The Mixed Tasks were intended to test whether: (a) participants had a preference for one mode of interaction over another and (b) whether participants could distinguish between tasks that were more suited to one mode than another. Consequently, Mixed Tasks were always done last, to ensure participants were already familiar with both Table Mode and Value Mode.

The tasks in both Value and Table Mode were organized as follows. The first two tasks required the participant to define only single rules or create only a simple, two-way table. These were followed by four progressively more difficult tasks. In Value Mode, the number of rules required for each task was increased and two tasks required the use of negative rules - where a member was excluded from the population. In Table Mode, the number of dimensions required for each task was progressively increased. In both modes, tasks were presented in such a way that more advanced OLAP functions provided a faster method of moving from one task to the next. For example, participants could go from one Table Mode task to another by simply removing a dimension rather than by starting a new table and adding all but one of the dimensions used in the previous task.

In the first six tasks, in both Table and Value Modes, participants were told which data set to use. However, the seventh task in both modes did not provide participants with the name of the required data set. Instead, the task gave a list of dimensions that would be required to complete the task. This meant participants needed to locate an appropriate data set.

Mixed Tasks were structured so that the first and third tasks were more easily accomplished in Value Mode, while the second and fourth tasks were more easily accomplished in Table Mode. This was achieved by requiring the participant to locate only one value in the first and third Mixed Tasks, but to locate two and three values in the second and fourth Mixed Tasks. This organization of tasks gave participants the opportunity to select a mode either on the basis of personal preference or because of perceived efficacy.