Experiment Design

---

As discussed in research method section, we decided to carry out three kinds of evaluation of our NetPay-enabled e-newspaper prototypes to determine their suitability for providing e-tailing system payment support:

• A usability evaluation surveyed users of the prototype to assess their impressions of the approach when carrying out information purchasing tasks using a micro-payment vs. a macro-payment protocol.

• A performance evaluation assessed the performance of NetPay-enabled web-sites to determine the overhead of the micro-payment extensions made to the software, particularly in regard to user response time.

• A qualitative evaluation assessed factors such as customer effort in using a NetPay-enhanced website from a customer's perspective, along with the cost/benefit of the system for customers, vendors and brokers.

In this section we summarise the design of each of these experiments, and in the following section report on the results of each experiment and draw conclusions from these as to the utility of a NetPay-based micro-payment system for an e-newspaper e-tailing application domain.

Usability Evaluation

We evaluated participants' user satisfaction, navigational efficiency, effectiveness and general preference for the three payment systems—subscription-based macro-payment, server-side, and client-side NetPay micro-payment with two newspaper sites (Preece, 1993; Rubin, 1994). These measures are the standard ones for determining how "usable" an interactive system is, and allow us to make judgements on the suitability of the interface for the tasks being carried out (Neilsen, 1992). Efficiency was measured by the degree of ease to change different newspaper sites and the speed of article content loading. Effectiveness was measured by assessing operations needed by the customer to complete their purchases. Satisfaction was a subjective measure assigned by each participant in the experiment.

We identified 10 participants to carry out a set of information purchasing tasks from our three e-newspaper prototypes: one using subscription-based macro-payment, one using a client-side NetPay e-wallet, and one using a server-side NetPay e-wallet. After completing these information searching and access tasks with each of these payment systems, participants answered a post-test questionnaire. We choose to use a task list and post-survey questionnaire rather than other usability evaluation approaches due to the ease of setting up the experiment, and also because we felt this would provide the best usability measures for such a problem domain (Preece, 1993).

After completing the tasks with all three payment systems, participants ranked the systems in order of preference. The application servers used are: newspaper1 and newspaper2 providing subscription-based macro-payment; broker, newspaper1, and newspaper2 providing server-side NetPay micro-payment; and broker, newspaper1, newspaper2 providing client-side macro-payment. These application servers were deployed for this experiment on the same host on the Windows XP network. The participants in the experiment used other PCs connected to this network to carry out a set of tasks including registering, subscribing, buying coins, reading articles and reading articles over multiple sittings.

Performance Impact/Evaluation

Our three prototypes providing subscription-based payment, server-side and client-side NetPay micro-payment have been tested for application server performance and client response time. The key aim was to test how long a newspaper site takes to serve client requests when extended to use each of the three payment systems from the time the customer clicks the title of an article to the time the article is fully displayed on screen. In order to do this we developed a pseudo-web browser to perform large numbers of requests to the web server and to time the response time of the web server. The macro-payment subscription-based approach makes one expensive macro-payment debit to pay for the initial subscription and then simply checks whether a customer, after login, has a valid subscription. The micro-payment systems need to carry out an e-coin debit of the customer's e-wallet with each purchase of an article. We measured the CPU time taken by the vendor's web server and the overall time taken to activate the page display. The longer the delay to display a page, the more problematic for the customer in terms of vendor information response time. The more CPU consumed by the server-side, the less overall client requests and lower response time overall can be supported.

Qualitative Comparison to Macro-Payment

We carried out a third, qualitative assessment of our three prototype e-newspaper websites to assess various factors associated with their costs and benefits for customers, vendors and the broker organization. The assessment criteria included:

• Number of customer interactions with the website(s) needed to read articles

• Information retention needed by customers to use website(s)

• Cost to customers depending on subscription and article pricing and article usage

• Cost to vendors of subscription authorization and e-coin redemption

• Cost to brokers of providing e-coins and redeeming coins with banks

The results for these analyses were obtained from analyzing the performance of each payment method in order to satisfy a payment scenario.

---