Active Experimentation

A marketing theory, for example, claims that a blue version of a Web page will outperform a red Web page on a particular criterion. To test this theory, when a visitor requests the Web page the server invokes a random number generator and by chance sends the visitor either the blue or the red version of the page. This methodology extends to two versions of an entire site with dozens of differing sub pages, or reduces to an experiment with two pages differing only in the wording on one link.

Experimenting on actual Web sites creates many advantages for the researcher. For one, using an actual site tends to lead to practitioner implications and managerial relevance. For another, the external validity is high as the sample consists of actual visitors to that site and the experiment is unobtrusive to those visitors (Dr ze & Zufryden, 1997). Internal validity can be high as well, because the sequence of events can be computerized and subject to random assignment. To cap it off, even modest sites can generate hundreds of cases in a few days. Once the researcher sets the experiment in motion, the data collect themselves .

The ease with which one can generate large samples of visitors on a site gives researchers time to investigate generalizing across multiple sites. That is, the research investigating red vs. blue pages might employ a variety of pages sampled from the universe of possible Internet pages. Presumably, the actual page used could be a random effect in the model used.

Technical Methods for E-Mail and Web Experimentation

Several mechanisms help execute experiments similar to those described in the preceding paragraphs. A CGI script (available from the authors) that generates the random number can log the page version sent and the visitor's IP address. After identifying unique visitors to the site, the log generated by the CGI script can then classify those visitors according the page version they saw.

A second mechanism that does not rely on a CGI script uses a non-existent 1 —1-pixel image embedded in the HTML code for each version of the page. The name of this image would vary according to the page version, for example blue.gif and red.gif. The log records will show a Return Code of 404 for blue.gif each time the server sends a blue version of the page, and similarly for the red page version.

Using this nonexistent image, one can forego using a random number generator. Changing the page version based on the time of day also creates experimental control. For example, assuming that the page in question is x.html, and that different versions of the page are a.html and b.html, one could execute a background program that runs, say, every 20 minutes and does this:

copy a.html to x.html

copy b.html to a.html

copy x.html to b.html

In effect, two versions of the page overwrite x.html every 20 minutes. One could also execute this code one extra time at midnight. Thus, each version has a different 20-minute interval from the day before, thereby counterbalancing the experimental design. Furthermore, using a different nonexistent image as discussed above - blue.gif vs. red.gif - helps track which visitor received which page version without worrying about calculating a visitor's particular 20-minute interval.

An embedded image can also track individual visitors in order to analyze aggregate choice probabilities. For a study that focuses on the click-through rate of one or more links on the experimental pages, one divides the number of people who clicked on a link by the number of people exposed to the links. A unique (nonexistent) image on each page version shows the denominator for the page versions via the log count of 404 return codes.

For the numerator, one sets up each version of the page with unique URLs. For example, to compare red and blue versions of a page, with the criterion variable being the number of clicks on the link to 'www.experiment.com/get-more- info .html,' the coded URL embedded in the red and blue versions of the page could be

www.experiment.com/get-more-info.html?red

www.experiment.com/get-more-info.html?blue

The Web server ignores anything after the question mark, as the hypertext transfer protocol ignores that part of the URL. But the Web server does log get-more-info.html?red and get-more-info.html?blue as two different pages, thereby recording the number of clicks on that link made from each page version. Dividing these numbers by the count of the non-existent images - from any standard Web log report or one available from the authors - yields the choice probabilities.

One studies e-mail messages similarly, giving the URLs embedded in the e-mail a unique code matched to the experimental condition. To test a blue page e-mail against a red page e-mail, the criterion variable is how many recipients click on a URL embedded in both emails, which points to the Web page, www.mycompany.com/newsletter-followup.html. The red and blue e- mails go out with these URLs, respectively:

www.mycompany.com/newsletter-followup.html?red

www.mycompany.com/newsletter-followup.html?blue

The words 'red' and 'blue' after the question mark permit simple log file analysis of which version yielded the most clicks. As with the Web page experiments, each e-mail version points to different nonexistent 1 —1-pixel images in order to estimate how many people in each experimental condition opened the e-mail. This number represents a lower limit as some mail programs let users turn off images or preview e-mail without opening the e-mail.

Possible Pitfalls

Occasionally a visitor may observe multiple experimental conditions by using the reload button on the experimental page. Drez and Zufryden (1997) suggested another randomization mechanism to avoid this possibility, namely to assign visitors to conditions using the last digit of the visitor's IP address. This approach helps assure that each visitor sees only one version of the site during the experiment. The down side is that proxy servers, mentioned earlier, make it problematic to assume that all users have the same IP address - even in a short site visit. One could avoid this pitfall, however, using cookies.