4 Experiment


4 Experiment

4.1 Subjects and method

25 subjects took part in this experiment. These subjects were interested in the Web and new technologies. They all regularly used the Internet. The experiment was performed using a laptop computer equipped with a mouse. Searches were performed using the Voil   portal.

4.2 The task types

Using the values of our four variables , we developed twelve types of task resulting from an almost complete crossing of the modes of our variables: repetitive/non-repetitive; explicit/implicit; local/distributed; unique/multiple. We performed all the possible crossings. Below is an example task and the accompanying description.

Task 1. Find a way of getting from Toulouse to Montpellier by public transport this Friday (arriving between 4 and 5 pm).

  • The number of targets variable here is "unique" since the only way of getting from Toulouse to Montpellier by public transport on this date and at this time is by train. The timetable is located on a single page in a single site, namely that of SNCF (the French railway operator).

  • The target distribution variable here has the value "local" since the timetables are all on the same page in the SNCF site.

  • The target explicitness variable here has the value "explicit" since the subject did not need to produce inferences or look for any other information in order to know whether or not there was a train matching the times requested .

4.2.1 Protocol

The subjects completed two pre-tests, one concentrating on their Web browsing skills and the other relating to their knowledge in the various fields.

The 25 subjects were randomly divided into two subgroups, one consisting of 15 subjects and the other of 10 subjects. The group of 15 subjects performed the non-repetitive tasks while the group of 10 subjects performed the repetitive tasks. In the group of subjects performing the non-repetitive tasks, each subject performed three tasks, namely one autonomous task and two prescribed tasks . In the other group , each subject performed one autonomous and one repetitive task followed, if time permitted, by a second repetitive task. The subjects started their searches from the Voil   portal interface and performed the tasks on the basis of the instructions given to them. Each of the non-repetitive tasks was performed by five subjects and each of the repetitive tasks was performed by two subjects. While the task was being performed, the path taken by the subjects was stored in order to keep a record of their search. Their comments were recorded and the time was measured.

4.2.2 Instruction

"You will see the Voil   portal interface and you must search for the information that you will be asked to find from this portal. After that, you will be asked to fill in a questionnaire concerning these searches.

First of all, what is the last information search that you have performed on the Internet at your own initiative? Could you repeat it, describing aloud what you did, i.e. what you clicked on and what you entered?

Now you are going to perform the search tasks that I am going to ask of you. You will start each task from the Voil   portal. For each defined task you will have to reformulate what is asked of you (and tell me how you intend to go about it)."

4.3 Results

We adopted an exploratory, qualitative approach in order to define our research hypotheses. For this reason, the results below are essentially qualitative and we have not performed any significance tests.

Our intention is to determine the effect of our four variables: "level of explicitness of the target", "quantity of targets", "distribution of targets" and "task repetitiveness" on the subjects' behaviour in an information-seeking task. To do this, we used the recall and precision indices.

In order to analyse the results, we attempted to develop an optimal search model for each of the search tasks. This model was to act as a reference when evaluating the searches performed by the subjects. We did not define information search models for the two search tasks: "repetitive, multiple, distributed and implicit" and "non-repetitive, multiple, distributed and implicit" since the number of equivalent possibilities was too great. The results obtained from the subjects on these tasks were not therefore taken into account in the calculation of the recall and precision indices. Nevertheless, an observation of the subjects during their search operations together with a qualitative analysis of the results allowed us to note that recall and precision appear to be at their weakest for these tasks. That is why it should be pointed out that the results below for the implicit, distributed and multiple tasks are probably better than the actual results.

4.3.1 Recall

Table 20.1: Mean recall for each value of each variable and difference in mean recall between each mode of each variable

Variables

Mode

Mean recall

Difference between the modes of each variable

Quantity of targets

unique

0.59

0.04

multiple

0.55

Distribution of targets

local

0.6

0.16

distributed

0.44

Explicitness of the target

explicit

0.66

0.23

implicit

0.43

Repetitiveness of the task

repetitive

0.72

0.22

non-repetitive

0.5

The recall index is the ratio of the number of targets accessed by the subject to the number of existing targets.

4.3.2 Precision

The precision index is the number of targets accessed by the subject divided by the number of pages opened by the subject.

We observed a greater or lesser effect of the variables on user 's activities. The variables which had the greatest effect on task success were also those that had the greatest effect on the precision of the information search, namely "repetitiveness of the task" and "level of explicitness of the targets". We observed that users performed repetitive and explicit tasks successfully and accurately. In contrast, the non-repetitive, distributed and implicit tasks were performed with difficulty and inaccurately.

Table 20.2: Mean precision for each value of each variable and difference in mean precision between each mode of each variable

Variables

Mode

Mean precision

Difference between the modes of each variable

Quantity of targets

unique

0.39

0.06

multiple

0.45

Distribution of targets

local

0.44

0.05

distributed

0.39

Level of explicitness of the target

explicit

0.49

0.15

implicit

0.34

Repetitiveness of the task

repetitive

0.64

0.3

non-repetitive

0.34

4.3.3 Effect of the different task types on users' activities

Our intention was to study the effect of crossing the modes of the variables on users' activities. However, it was difficult to study the results of the "repetitive" tasks given that only two subjects were tested for each of these tasks. The difference between the "repetitive" tasks and the "non-repetitive" tasks is due to the fact that one was performed just once whereas the other was performed a number of times. We can therefore hypothesise that the conclusions that we are able to draw from the "non-repetitive" tasks will also apply to the "repetitive" tasks if we consider that, generally speaking, these will be characterised by higher recall and precision indices.

The figure below represents the effect of crossing the variables in the non-repetitive tasks on subjects' activity.

click to expand
Figure 20.1: Mean recall and precision for each task type across all the subjects (the position of the "implicit, multiple, distributed" task is indicated in grey; our hypothesis is that this task should have lower recall and precision indices than the other tasks)

We can observe that, overall, as recall increases from one type of task to another, precision also increases and vice versa. The tasks for which recall and precision are high are the tasks in which the subjects made few errors and found a large number of targets. In contrast, the tasks for which these two indices were low were the tasks in which the subjects made a large number of errors and found only a few targets. This relation between recall and precision is unusual. In fact, the usual relation between these two indices is of the type: precision = 1 “ recall (inversely proportional) [BUCKLAND, 1994]. In other words, generally speaking the broader the search conducted by the subjects and the larger the number of pages opened, the greater the likelihood that a large number of targets will be accessed. [BUCKLAND, 1994] describes the results we have obtained as "perverse": precision increases with recall. Everything suggests that, depending on the nature of the task, all the subjects either manifested imprecise behaviour in opening a large number of pages irrespective of their content or precise behaviour consisting of opening only the relevant pages.

One approach that would permit us to develop interesting hypotheses concerning subjects' activities would be to find a way of predicting the effect of each of the variables on each task type. To do this, we decided to arrange our variables hierarchically as a function of their effect on recall and precision

4.3.4 Hierarchical organisation of the variables

We used a contrastive approach to perform this task classification. First of all, we calculated the variable that had the greatest effect (all other variable values confounded) and we then arranged the two modes of this variable in hierarchical order on the basis of their recall and precision scores. We repeated this operation for each value of each variable, thus gradually reducing the number of variables for comparison. We then applied the results obtained for the non-repetitive tasks to the repetitive tasks. This results in a tree structure with the variables arranged from left to right as a function of their effect on recall and precision and the tasks arranged from top to bottom as a function of their recall and precision levels.




Communicating With Smart Objects(c) Developing Technology for Usable[... ]stems 2003
Linux Troubleshooting for System Administrators and Power Users
ISBN: N/A
EAN: 2147483647
Year: 2005
Pages: 191

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net