The Experiment
This experiment had two fundamental objectives: (1) to replicate the finding that skill acquisition leads to a preference for an incumbent Web site (Johnson et al., 2003; Murray and H ubl, 2003); and (2) to test the impact that experience with an incumbent interface has on consumers' perceptions of that Web site interface relative to its competitors . The study was fully computer-based, and involved the completion of an online shopping search task. The entire experiment was conducted in an Internet-based electronic store adapted from Murray and H ubl (2002). This study was completed for course credit by 54 undergraduate psychology students in a research laboratory equipped with state-ofthe-art networked personal computers. The experimental design was balanced with 27 respondents in each condition, all of who participated via a secure local area network (LAN) and were randomly assigned to one of two experimental conditions (see below).
The data analysis for this experiment is separated into two parts . First, I analyze the relative choice share data, as well as the learning data (mean task completion times, and the standard deviations of task completion times across subjects). Second, I examine the impact that practice has on the rating-scale measures (Table 6-1).
Method
-
Experimental Design. This experiment is a between-subjects single factor (amount of practice with the incumbent interface) design. The amount of practice was manipulated at two levels: participants are randomly assigned to experience either one or six trials with the incumbent before being exposed to the competitor.
-
Procedure . Those subjects who completed only one shopping task with the incumbent were required to complete filler tasks so that all subjects completed six tasks in total. Each shopping task required subjects to find a product by searching within a product space that contained 16 products (see Table 6-2 for an example product category).
Table 6-2: Sample product category (Apple iMac) Attribute 1
Attribute 2
Attribute 3
Attribute 4
Attribute 5
Color
Processor Speed
RAM
Internet Browser
Price [2]
Sage
500 MHz G3
32 MB
Netscape Communicator
$2,347.00
Ruby
500 MHz G3
64 MB
Microsoft Internet Explorer
$2,448.00
Snow
500 MHz G3
64 MB
Netscape Communicator
$2,498.00
Indigo
500 MHz G3
64 MB
Microsoft Internet Explorer
$2,647.00
Indigo
450 MHz G3
32 MB
Microsoft Internet Explorer
$1,999.00
Sage
450 MHz G3
64 MB
Netscape Communicator
$2,148.00
Ruby
450 MHz G3
64 MB
Microsoft Internet Explorer
$2,198.00
Snow
450 MHz G3
64 MB
Netscape Communicator
$2,299.00
Snow
400 MHz G3
32 MB
Microsoft Internet Explorer
$1,547.00
Indigo
400 MHz G3
32 MB
Netscape Communicator
$1,648.00
Sage
400 MHz G3
32 MB
Microsoft Internet Explorer
$1,698.00
Ruby
400 MHz G3
64 MB
Netscape Communicator
$1,847.00
Indigo
350 MHz G3
32 MB
Netscape Communicator
$1,199.00
Sage
350 MHz G3
32 MB
Microsoft Internet Explorer
$1,348.00
Ruby
350 MHz G3
32 MB
Netscape Communicator
$1,398.00
Snow
350 MHz G3
64 MB
Microsoft Internet Explorer
$1,499.00
[2] All prices are in Canadian Dollars
To assist them, participants had access to a screening tool that allowed them to specify particular product attributes and thereby reduce the number of products through which they would have to manually search to complete the task. In order to complete the experiment, participants were required to successfully complete each assigned task. As a result, they were motivated to learn to complete the tasks accurately and efficiently . However, it is likely that for many real world utilitarian tasks, consumers are more involved in the purchase and are driven by more powerful motives, which makes the results of this experiment conservative in this regard.
The following is an example of one shopping task: 'Your task is to select the Apple iMac with the fastest processor, given that it is under $2,000.00 and is snow colored.' Which interface served as the incumbent was counterbalanced between conditions (for example, the condition wherein the screening tool with the pull-down menus served as the incumbent is illustrated in Figure 6-1). Those participants who did not complete all six shopping tasks were instead exposed to filler tasks based on the same product category as the shopping task that they did not complete. However, the filler task was not search- related and instead required subjects to simply list product attributes and to estimate the product's average price.
Figure 6-1: Sample interface screen shots
After completing the shopping task either once or six times with the incumbent interface subjects were forced to use a competitor (i.e., different) interface to complete a seventh shopping task. Having completed the seventh task using the competitor, subjects were asked to choose which of the two interfaces they would like to use to complete one more shopping task. In addition to choosing between the two interfaces subjects were asked to indicate how much they preferred the interface that they had chosen on a scale from 1 (just barely prefer) to 10 (very strongly prefer). Having made a choice and indicated the extent of their preference, participants completed one more shopping task using the interface of their choice. After completing the last shopping task the participants completed a series of rating-scale responses (Table 6-1).
Results
Skill Acquisition and the Development of Preference. Figure 6-2 illustrates the learning curve of participants in the six-trial condition. It is clear from this figure that a substantial amount of learning has occurred between the first and sixth trial as the mean task completion times decrease from approximately 76 seconds on the first trial to approximately 24 seconds by the sixth trial. Moreover, the standard deviation of the task completion times across participants has fallen from approximately 68 seconds on the first trial to only seven seconds by the sixth trial (Figure 6-3).
Figure 6-2: Task completion times (six trial condition)
Figure 6-3: Standard deviations of the task completion times (six trial condition)
As predicted by the research on the power law of practice (Johnson et al., 2003; Logan, 1992), both of these curves are well approximated by a power function (task completion times R 2 = 0.918; standard deviations, R 2 = 0.740). In combination these two pieces of data indicate that a considerable amount of skill acquisition has taken place that allows for significantly faster task completion times (t-test of task completion times between trials 1 and 6 within the six trial condition: M = 76.3 seconds & 24.0 seconds; p-value < 0.0001), and that as a group participants are converging towards a minimum task completion time.
Next , I look at the relative choice shares for the incumbent interface between the two levels of practice (Figure 6-4). While initially, there is no difference in the choice shares between the incumbent and the competitor (based on a binomial test, using equal choice shares as the null hypothesis, p-value = 0.351), by the sixth trial there is a strong preference for the incumbent (again using a binomial test with equal choice shares as the null hypothesis, p-value = 0.003).
Figure 6-4: Incumbent choice share by the number of incumbent trials
To model the impact that the number of tasks has on the participants' preference for the incumbent, a logit model is employed, with choice of interface (incumbent or competitor) as the dependent variable and the number of tasks completed with the incumbent before exposure to the competitor as the independent variable. The results indicate that the number of tasks completed with the incumbent before exposure to the competitor had a significant effect on participants' choice of interfaces ( c 2 = 5.445, p-value = 0.020).
Although preference data were collected in addition to the choice data, I have modeled the choice shares alone because of the extreme non-normality of the preference data (Figure 6-5). Nevertheless, the bi-modal distribution of the preference data does indicate that, in general, participants have a strong preference for the interface that they chose.
Figure 6-5: Distribution of graded pair comparison (preference) ratings
The above analysis provides compelling evidence that participants are acquiring skills that allow them to complete the task more rapidly with additional trials and that with additional trials a preference develops for the incumbent interface.
The Effect of Practice on Perception. The next step in our analysis is to look at the impact of practice on participants' perceptions of the two interfaces. A separate t-test was run for each of the key rating-scale responses (Table 6-1), comparing ratings between the two levels of practice (either 1 or 6 incumbent trials). The results indicate that only perceptions of the quality of the incumbent interface (question 1) change significantly, such that as experience increases so do perceptions of quality (M = 6 & 7.11; t-test p- value = 0.035). The difference in ratings of emotional attachment to the incumbent (question 3) between the two conditions is marginally significant (M = 2 & 3.33; p-value = 0.094), while the difference between conditions for all other rating scales is not significant (all p-values > 0.400). An analysis of variance (ANOVA) testing the differences in quality ratings for the incumbent (question 1) and the competitor (question 2) in both the 1 and 6 trial conditions (Figure 6-6) is significant (F = 2.692; p-value = 0.013). Follow-up t-tests indicate that while there is no difference between quality ratings for the competitor interface across trial conditions (p-value = 0.728), and there is no difference in quality ratings between the competitor and the incumbent in the one trial condition (p-value = 0.590), there is a significant difference between the competitor and the incumbent in the six trial condition (p-value < 0.002).
Figure 6-6: Incumbent and competitor ratings of interface quality by trials
The final section of the analysis examines the impact that interface perceptions (the ratings-scale questions 1 through 6) have on choice. Each of the rating-scales is included as a predictor variable in a logit model with choice as the dependent variable. The only rating-scale variable that has a significant effect on choice at conventional levels is question 1, the perceived quality of the incumbent interface (t-value = 2.130; p-value = 0.034). With the exception of question 2 (quality of the competitor), which has a marginally significant effect on choice (t-value = -1.707; p-value = 0.091), all other rating-scales have no effect on participants' interface choice (all p-values > 0.350).
Discussion
The results from this experiment replicate previous findings (Johnson et al., 2003; Murray and H ubl, 2003) by demonstrating that with practice learning does occur and that with increasing task experience the probability that participants will choose the incumbent increases. However, in addition to replicating previous findings, this experiment was designed to examine the impact that experience has on a number of perceptual measures. The results indicate that with increasing experience, participants' perceptions of product quality increase. Although the two interfaces are not perceived to be any different when each has been used one time, there is a significant difference in quality perceptions between the interfaces when the incumbent has been used six times and the competitor has only been used once. In addition, while there is a significant difference in incumbent quality perceptions between the one and six trial conditions, there is no difference in competitor quality perceptions in the one and six trial conditions.
It is worth noting that perceptions of quality have an impact on the choices that participants make. Therefore, changes in perception that occur with increasing exposure to the incumbent are meaningful and can have an impact on an interface's market share. In sum, these data provide strong evidence that perceptions of interface quality are affected by experience with an interface in a way that gives the incumbent interface an advantage over competitors.
EAN: 2147483647
Pages: 164