Modelling the Task of Retrieving and Analysing ESD

Based on the pilot studies, conducted by Hyland and Hasan (1997) and by Hyland and Gould (1998) and on the results of usability testing in the current study, a more complete, empirical model for the task of ESD use by casual users can be offered and is shown in Figure 4.

Figure 4: A Proposed Model for ESD Use Based on Goodhue's Model (1998)

The model, shown in Figure 4, does not attempt to describe the use of ESD by experts or how ESD might be manipulated by the entire range of available SIS, e.g., spreadsheets, statistical packages, GIS. Instead, it attempts to model the type of analysis and retrieval tasks that might be carried out by casual users using the less complex functions available in some SIS, including OLAP, obviously.

The model concerns the task of retrieving and analysing ESD, regardless of the type of SIS used to do so. Because of this, the terms used to describe OLAP systems (e.g., dimensions, members) are no longer appropriate. Instead, the model uses the terms "variables" and "values" to describe the same concepts. These terms are used in the statistics literature and in some of the more popular statistical packages. They might not be recognized in statistical database or GIS circles but the usage is reasonably common. The Abacus prototype used the terms "variables" and "values" rather than "dimensions" and "members" for the same reasons.

The location sub-task in the proposed task model involves not only finding one or more sets of ESD but also judging the suitability of those sets of data. In a real life situation, a user may have to decide whether a particular source of ESD is reliable, whether the data is sufficiently current and whether the values in the data are likely to satisfy the user's information needs. In the model proposed by Goodhue (1998), these decisions are dealt with in a separate sub-task at the end of the process. When dealing with ESD, it does not appear that this is a suitable location for these decisions to be made. The organization sub-task is a complex one and it would seem to be foolhardy to spend time and effort organizing ESD that was not sufficiently current or reliable for one's information needs. Consequently, the decision about the suitability of ESD would be better made prior to the Organization sub-task. In practice, however, it is possible that some or all of the sub-tasks will overlap, as originally suggested by Goodhue. It is quite probable these judgemental sub-tasks would also be carried out following the organisation sub-task, as well.

In addition to these considerations, the location sub-task addresses the issue of finding a set of data that contains the required variables and values of those variables, i.e., a user would have to ascertain if a data set contained all of the variables that he or she required and if so, whether the necessary values for each those variables were available. In some SIS, it might be possible to calculate new variables or to calculate new values or group certain values together. If this were the case, then a data set that did not appear to have the required variables or values might be modified to produce the required ones and thus satisfy the user's needs. The user would need to take this possibility into account when selecting the data to be used.

Because the user observation carried out in this study is concerned more with the mechanical aspects of the location sub-task rather than the judgmental aspects described above, little additional detail can be provided about those judgemental tasks at this time. The location sub-task will be described in more detail in a later section of the chapter.

The organization sub-task involves four steps that could be carried out serially or in parallel. Firstly, the user must identify the required variables and include only those variables in the display, i.e., table, chart, map. This could involve the addition of variables to the display or the removal of unwanted variables from the display. It may be necessary at this time to calculate new variables (e.g., combine two or more variables, etc.). The user must then assess the suitability of the default level of aggregation and either zoom in or zoom out on one or more values of the chosen variables. Once again, the user may need to calculate new values or group values together. The final step is the placement of variables and values in such a way as to highlight any features of the data that the user thinks are important. In the case of a table, for example, this might involve locating one or more variables as rows while other variables are placed as columns. In cases where this resulted in multiple rows or multiple columns, the placement of variables in that hierarchy would also be significant. These appear to be the major steps in the organization sub-task in a variety of SIS. These will be discussed in greater detail in three later sections of the chapter.

The presentation sub-task consists of two parts: selecting the display type and the display style. The display type might be a table, chart or map. The decision as to which type of display to use would depend on the amount of data, the level of detail that is necessary and the audience and purpose for which the display is intended. Charts and graphs may be more helpful in showing trends while tables provide more detail. Obviously maps are only appropriate where a geographical dimension is important. The display style would include decisions about fonts, colours, shading, cross-hatching and so on. These can be important factors in determining the usefulness of a display of ESD. Although the Abacus prototype supports both tables and charts (but not maps), the tasks completed in usability testing did not involve the use of different display types or the incorporation of such displays into reports or other documents. So, no further details about the presentation sub-task can be determined from the current study.

Details of the Location Sub-Task

One of the aims of user observation was to gather data about the manipulation of ESD by participants and so the tasks were designed so that participants could quickly locate the required data set and begin manipulating it. This was achieved, by telling participants which data set they required, for 16 of the 18 tasks, and simply having them select those data sets. Because the process of locating data sets was also of interest, two tasks were included in which participants needed to locate an appropriate data set. For these two tasks, participants were not told the name of the data set that they needed to use, only which dimensions would be required along with a passing reference to the type of data that would be needed, i.e., whether the data was about people, families or dwellings. The tasks had identical formats and were located at comparable positions in relation to the other tasks. One was the last task in Value Mode and required the Dwellings A data set, the other was the last task in Table Mode and required the People B data set.

While these two tasks provide an opportunity to study some aspects of the location sub-task, they are only concerned with the process of ensuring that required data exists in a set of ESD, not about the evaluation of the set of ESD, itself. Participants were certainly never required to make judgements about the currency, accuracy or reliability of the data they were using, as they may have to do in real life situations.

The times taken for each location task were recorded for all participants. The results were quite surprising. In the simpler of the two location tasks, participants only needed to look in the first of the sets of Dwelling data, i.e., Dwellings A, to locate the correct data set. Because the task was relatively simple, the minimum time taken to locate Dwellings A was only four seconds and 30 percent of participants completed the task in 10 seconds or less. Surprisingly, 33 percent of participants had great difficulty with the location process, taking a long time to complete the tasks and making many errors during the process. The maximum time taken to complete the first location task was 286 seconds. Even after removing the five longest times, 27 percent of the participants took between 30 and 90 seconds to complete the location task. The behaviour of participants was often quite illogical. For example, some participants ignored the reference to Dwellings in the task, and began by inspecting the People data sets. Other participants repeatedly inspected data sets Dwellings B and Dwellings C but did not inspect Dwellings A.

It should be remembered that the actions being described are only the simplest aspects of the entire location sub-task and only require the participant to confirm the presence of relevant dimensions. The location sub-task would normally require the participant to confirm the presence of appropriate members of those dimensions as well. For example, a user who wanted to know the number of plumbers in a region would not be satisfied with a data set containing an occupation dimension if the lowest levels of aggregation for occupation were tradesmen, managers and professionals. The process of locating a suitable data set would typically require the user to confirm the presence of a number of members for a number of dimensions. Given the difficulty in completing the simple location task in this study, it can be assumed that the more complex tasks of matching members as well as dimensions would be problematic for many users. Moreover, the location sub-task also involves more complex activities such as the evaluation of currency, accuracy and reliability of the data sets. When viewed in its entirety, it appears that this is an extremely complex process.

Based on the data gathered by means of user observation and reflection on the development of the prototype, a normative model of the location sub-task has been developed and is shown in Figure 5. Since the data gathered by means of user observation does not concern the evaluation parts of the location sub-task, the following model deals only with the mechanical aspects of the location sub-task. The terms "variables" and "values" have been used, as these terms are applicable in a variety SIS and would be more widely understood than the terms "dimensions" and "members," which are associated specifically with OLAP and MDDB.

Figure 5: A Proposed Model for the Location Sub-Task

The model provides a recommended set of steps for casual users to follow in the location sub-task. In practice, many users would probably vary this process, by overlapping the location and organization sub-tasks. For example, as soon as a user had located a required dimension, he or she might add the dimension to a table or chart before locating other required dimensions. This is not as efficient as the model proposed above, which ensures that all required dimensions are available before adding any dimensions to the table or chart.

Strategies Used in the Organization Sub-Task

One of the objectives of usability testing was to find out how participants would use the various OLAP functions to complete tasks. Both the training materials and the tasks were deliberately designed to encourage participants to evolve strategies to complete their tasks. Participants did use a significant number of strategies to complete the tasks. Several of these were either unhelpful or idiosyncratic and will be omitted from further discussion.

Strategies Used in Value Mode

Because of the highly structured nature of Value Mode, only one real strategy was used. The process followed was:

• To clear any existing rules (in one of several ways).

• Then, repeatedly select Define a rule.

• Select a dimension to which the rule would apply.

• Select a member from that dimension.

• Either include or exclude that member from the total being displayed.

This approach was used at least once by all 31 casual participants. Because it was implicit in the organization of the interface, it was used extensively, a total of 265 times in all. Little can be learned from this strategy about the way users might carry out the organization sub-task because the strategy to be used is pre-determined by the interface itself. It can be said, however, the strategy used in Value Mode is one that users appear to understand and can adopt quite quickly. In itself, this is useful information.

Strategies Used in Table Mode

Only two fundamentally different strategies were observed in Table Mode. However, there were a number of interesting variations in the second of these strategies and also in the way participants moved from one task to the next. Each of these variations is indicative of the way the participants were able to employ the available OLAP functions.

In Table Mode, the first fundamental strategy was:

• To clear any existing table.

• Then, repeatedly apply slices, one for each dimension described in the task, until the required value is shown on the table.

This strategy does not result in a true table being formed and so participants who used this method were actually in violation of some of the instructions on their task sheets. This strategy was used at least once by ten participants (27.8 percent) and was used to complete a total of 47 tasks. This approach is very fast and many of the participants who used it completed the tasks very quickly as a result. Unfortunately, it is also quite limited in its applicability because it cannot be used when there are two conditions for a single dimension e.g. finding the total number of people who are either widowed or divorced. Despite its limitations, it is a very effective strategy and should be used wherever possible. However, users need to understand its limitations and the use of other strategies to overcome those limitations.

The second fundamental strategy in Table Mode was:

• To clear the existing table.

• Then, repeatedly add a dimension as a row or column.

• Zoom in to those dimensions to the required level of aggregation.

This strategy was used at least once by 32 of the participants (88.8 percent). In its simplest form, it was used 167 times. This approach often resulted in a very large table that participants found difficult to use. Faced with such large tables, participants often had to scroll repeatedly backwards and forwards through the table, keeping track of their position by putting their finger on the monitor screen. Despite this difficulty, many users never evolved any other strategy for controlling table size. There were, however, many variations on this method that did solve the problem of table size by combining this fundamental approach with other OLAP functions. These will be described in a moment.

This second Table Mode strategy is only effective for simple tables containing one row and one column. In many cases, users might only require such simple tables, in which case, this strategy will serve them well. Table size could still become a problem if one or both of the dimensions on the table had a large number of members. In such cases, or when dealing with more complex tables, one of the variants of this strategy would be far more efficient.

There were three successful variants of the second strategy in Table Mode:

1. Do Strategy 2, then, limit the size of the table by defining a rule.

2. Do Strategy 2, then, slice one or more variables not on the table.

3. Add one or more variables to a table then combine slicing, defining rules and zooming.

Having described the strategies and their variations, it is timely to make two comments. First, the majority of participants always used strategy two, without any variation when using Table Mode. In fact, the variations on this strategy were only used on about 25 percent (N = 46) of the occasions when it would have been appropriate. Nonetheless, participants did develop significantly complex strategies even during their first encounter with Abacus. Second, it must be remembered the training materials only showed participants how to use the OLAP functions, not how to combine them with one another. Consequently, the evolution of the initial strategy and these three variants on that strategy demonstrates that casual users can integrate OLAP functions to produce desired results.

A second set of variations was observed in the way participants moved from one task to another. When moving from one task to the next, most participants would clear the existing table. The generation of a new table at the start of a task may appeal to participants because it returns them to a stable point from which they can follow the same strategy every time. It also ensures that they do not have any residual settings in place from a previous task. However, the tasks themselves were organized in such a way that it was, at times, very inefficient to start a new table. Instead, it was preferable to use some of the OLAP functions to simplify the movement from one task to the next and preserve much of the current table.

The variations on this process included adding new dimensions to an existing table, removing an unwanted member from an existing table or slicing a new dimension rather than adding it to the table. These variations are not necessary to complete the tasks, only to reduce the time taken to complete all of the tasks. It might have been expected casual users would either lack the logical or organisational skills to use such complex variations or be too involved with getting the answers correct to concern themselves with such efficiencies. Judging from the number of such variations developed and the number of individuals who made use of them, this is clearly not the case. For example, one variation was used by 24 percent (N = 7) of the casual participants. The use of these variations was both frequent and widespread.

Extending the Model of the Organisation Sub-Task

Based on the data gathered from usability testing, a normative model of the organization sub-task is proposed (see Figure 6). Since this analysis is not concerned with the location sub-task, the model assumes that all required dimensions and members of dimensions are available.

Figure 6: Proposed Model of the Organization Sub-Task Using OLAP

The model shown in Figure 6 is a normative model in that it represents a set of recommended steps for casual users to follow. Although it is based on a substantial body of empirical evidence, it does not attempt to model all the strategies observed. Instead it represents the results of critical reflection on various strategies and proposes one strategy that appears to produce the required results in a very efficient manner.

Unlike the proposed model of the location sub-task, which would be appropriate for a number of types of SIS, the model of the organization sub-task is only appropriate for SIS having analytical functions similar to those found in OLAP types of applications. The sorts of activities carried out with other types of SIS are extremely varied. For example, the calculation of standard deviations or linear regressions for a set of statistical data is quite a simple task in a statistical package but is not possible in a package such as Abacus and may not be possible in many OLAP environments. Similarly, the tasks carried out in a GIS or a spreadsheet might be quite different to those carried out in an OLAP environment. Given the enormous differences in functionality provided by the full range of available SIS applications, it does not appear feasible to produce a single task description for them all.

Concluding Comments on the Development of a Task Model

The data gathered by means of user observation have allowed us to identify and categorize some of the strategies that were employed by casual users when retrieving and analysing ESD. This, in turn, has allowed us to confirm the usefulness of the initial task model and to provide additional detail of some of the sub-tasks suggested in that model.

Although the results described above are interesting and useful, they are subject to a number of limitations. First, the strategies described have been evolved to cope with a specific set of tasks. These tasks may have elicited only a limited set of strategies or pre-disposed users to one strategy more than to another. For example, several users were able to answer all the tasks in Table Mode by using only the slicer. However, if any of the tasks had involved two constraints on the same dimension, this technique would have failed and those participants would have been forced to evolve a new strategy.

Second, the strategies above are directly linked to the use of OLAP pfunctions, with which the prototype has been built. A different set of strategies would be found if the prototype had used GIS, for example. Having said this, it is difficult to imagine how users could arrive at the desired result without having some mechanism to control the addition of dimensions (variables), the paring of unwanted members (values) or the aggregation or disaggregation of the data. With this in mind, the above models would still be applicable, to some extent, to other tools that provided similar functions.

Finally, the results of the study do not allow us to reach any conclusion about the efficiency or "learnability" of the strategies that have been evolved. It is possible that one of the strategies described above is more efficient, more intuitive or both, than any of the others. This might not become apparent during a user's first interaction with the prototype but only on subsequent interactions. Of even more concern is the possibility that other highly efficient and learnable strategies exist and that these were not evolved at all. It is a very real possibility that such strategies might be found by other analytical methods and simply taught to users. The normative model, shown in Figure 6, would appear to be one such strategy.

Limitations of the Research

This study has been exploratory in nature and as such it has some inherent limitations.

• The tasks carried out during testing do not cover the entire range of tasks carried out by casual users of ESD. For example, participants only needed to find a single numerical value but casual users frequently need to compare several values. Thus, the testing has not been exhaustive and the use of only one type of task may have skewed the results.

• Because of the amount of data being collected via video-recordings, it was decided to use only one protocol for the collection of data. This protocol does not provide any verbal clues as to the cognitive processes of the participants. As a result, we can only guess about the models of the task being used by participants. Other protocols, such as thinking-aloud, and post-event protocols could be used to provide information about such cognitive processes.

• The only data used in the evaluation of the prototype was census data. It is unclear whether the use of another type of data with which users were less familiar might have affected the performance of participants. Observation of users working with different data sets would be valuable to ascertain the extent of the difference in performance.