25 subjects took part in this experiment. These subjects were interested in the Web and new technologies. They all regularly used the Internet. The experiment was performed using a laptop computer equipped with a mouse. Searches were performed using the Voil portal.
Using the values of our four variables , we developed twelve types of task resulting from an almost complete crossing of the modes of our variables: repetitive/non-repetitive; explicit/implicit; local/distributed; unique/multiple. We performed all the possible crossings. Below is an example task and the accompanying description.
Task 1. Find a way of getting from Toulouse to Montpellier by public transport this Friday (arriving between 4 and 5 pm).
The number of targets variable here is "unique" since the only way of getting from Toulouse to Montpellier by public transport on this date and at this time is by train. The timetable is located on a single page in a single site, namely that of SNCF (the French railway operator).
The target distribution variable here has the value "local" since the timetables are all on the same page in the SNCF site.
The target explicitness variable here has the value "explicit" since the subject did not need to produce inferences or look for any other information in order to know whether or not there was a train matching the times requested .
The subjects completed two pre-tests, one concentrating on their Web browsing skills and the other relating to their knowledge in the various fields.
The 25 subjects were randomly divided into two subgroups, one consisting of 15 subjects and the other of 10 subjects. The group of 15 subjects performed the non-repetitive tasks while the group of 10 subjects performed the repetitive tasks. In the group of subjects performing the non-repetitive tasks, each subject performed three tasks, namely one autonomous task and two prescribed tasks . In the other group , each subject performed one autonomous and one repetitive task followed, if time permitted, by a second repetitive task. The subjects started their searches from the Voil portal interface and performed the tasks on the basis of the instructions given to them. Each of the non-repetitive tasks was performed by five subjects and each of the repetitive tasks was performed by two subjects. While the task was being performed, the path taken by the subjects was stored in order to keep a record of their search. Their comments were recorded and the time was measured.
"You will see the Voil portal interface and you must search for the information that you will be asked to find from this portal. After that, you will be asked to fill in a questionnaire concerning these searches.
First of all, what is the last information search that you have performed on the Internet at your own initiative? Could you repeat it, describing aloud what you did, i.e. what you clicked on and what you entered?
Now you are going to perform the search tasks that I am going to ask of you. You will start each task from the Voil portal. For each defined task you will have to reformulate what is asked of you (and tell me how you intend to go about it)."
We adopted an exploratory, qualitative approach in order to define our research hypotheses. For this reason, the results below are essentially qualitative and we have not performed any significance tests.
Our intention is to determine the effect of our four variables: "level of explicitness of the target", "quantity of targets", "distribution of targets" and "task repetitiveness" on the subjects' behaviour in an information-seeking task. To do this, we used the recall and precision indices.
In order to analyse the results, we attempted to develop an optimal search model for each of the search tasks. This model was to act as a reference when evaluating the searches performed by the subjects. We did not define information search models for the two search tasks: "repetitive, multiple, distributed and implicit" and "non-repetitive, multiple, distributed and implicit" since the number of equivalent possibilities was too great. The results obtained from the subjects on these tasks were not therefore taken into account in the calculation of the recall and precision indices. Nevertheless, an observation of the subjects during their search operations together with a qualitative analysis of the results allowed us to note that recall and precision appear to be at their weakest for these tasks. That is why it should be pointed out that the results below for the implicit, distributed and multiple tasks are probably better than the actual results.
Variables | Mode | Mean recall | Difference between the modes of each variable |
---|---|---|---|
Quantity of targets | unique | 0.59 | 0.04 |
multiple | 0.55 | ||
Distribution of targets | local | 0.6 | 0.16 |
distributed | 0.44 | ||
Explicitness of the target | explicit | 0.66 | 0.23 |
implicit | 0.43 | ||
Repetitiveness of the task | repetitive | 0.72 | 0.22 |
non-repetitive | 0.5 |
The recall index is the ratio of the number of targets accessed by the subject to the number of existing targets.
The precision index is the number of targets accessed by the subject divided by the number of pages opened by the subject.
We observed a greater or lesser effect of the variables on user 's activities. The variables which had the greatest effect on task success were also those that had the greatest effect on the precision of the information search, namely "repetitiveness of the task" and "level of explicitness of the targets". We observed that users performed repetitive and explicit tasks successfully and accurately. In contrast, the non-repetitive, distributed and implicit tasks were performed with difficulty and inaccurately.
Variables | Mode | Mean precision | Difference between the modes of each variable |
---|---|---|---|
Quantity of targets | unique | 0.39 | 0.06 |
multiple | 0.45 | ||
Distribution of targets | local | 0.44 | 0.05 |
distributed | 0.39 | ||
Level of explicitness of the target | explicit | 0.49 | 0.15 |
implicit | 0.34 | ||
Repetitiveness of the task | repetitive | 0.64 | 0.3 |
non-repetitive | 0.34 |
Our intention was to study the effect of crossing the modes of the variables on users' activities. However, it was difficult to study the results of the "repetitive" tasks given that only two subjects were tested for each of these tasks. The difference between the "repetitive" tasks and the "non-repetitive" tasks is due to the fact that one was performed just once whereas the other was performed a number of times. We can therefore hypothesise that the conclusions that we are able to draw from the "non-repetitive" tasks will also apply to the "repetitive" tasks if we consider that, generally speaking, these will be characterised by higher recall and precision indices.
The figure below represents the effect of crossing the variables in the non-repetitive tasks on subjects' activity.
We can observe that, overall, as recall increases from one type of task to another, precision also increases and vice versa. The tasks for which recall and precision are high are the tasks in which the subjects made few errors and found a large number of targets. In contrast, the tasks for which these two indices were low were the tasks in which the subjects made a large number of errors and found only a few targets. This relation between recall and precision is unusual. In fact, the usual relation between these two indices is of the type: precision = 1 “ recall (inversely proportional) [BUCKLAND, 1994]. In other words, generally speaking the broader the search conducted by the subjects and the larger the number of pages opened, the greater the likelihood that a large number of targets will be accessed. [BUCKLAND, 1994] describes the results we have obtained as "perverse": precision increases with recall. Everything suggests that, depending on the nature of the task, all the subjects either manifested imprecise behaviour in opening a large number of pages irrespective of their content or precise behaviour consisting of opening only the relevant pages.
One approach that would permit us to develop interesting hypotheses concerning subjects' activities would be to find a way of predicting the effect of each of the variables on each task type. To do this, we decided to arrange our variables hierarchically as a function of their effect on recall and precision
We used a contrastive approach to perform this task classification. First of all, we calculated the variable that had the greatest effect (all other variable values confounded) and we then arranged the two modes of this variable in hierarchical order on the basis of their recall and precision scores. We repeated this operation for each value of each variable, thus gradually reducing the number of variables for comparison. We then applied the results obtained for the non-repetitive tasks to the repetitive tasks. This results in a tree structure with the variables arranged from left to right as a function of their effect on recall and precision and the tasks arranged from top to bottom as a function of their recall and precision levels.