Experimental Results

---

Usability Evaluation

Ten participants volunteered for our usability study of NetPay. They were an equal mix of non-IT specialists and graduate students, the latter who were frequent users of online information portals. All participants were familiar with using e-tailing websites, particularly for purchasing books, CDs, and clothing. Participants were asked to complete five tasks with each system.

• Subscribe with the newspaper site or register and buy e-coins with a broker

• Read 3 articles on newspaper1 site

• Change to newspaper2 site and read 3 articles

• If subscription expired or e-coins run out, the user must renew it

• Read articles on the two vendor sites a second time, subsequent to the first use of the system

In the post-test questionnaire, we used a 5-point rating scale (1=Strongly Disagree, 5=Strongly Agree) to rate each tested characteristic. We also included open questions to gain user feedback to help in the qualitative analysis evaluation work. We presented the average ratings for the tested characteristics in a bar chart form as shown in Figure 7. The tested characteristics are:

1. Ease of use: Payment system is easy to use.

2. Efficiency1: It is easy to move around to different newspaper sites.

3. Efficiency2: The speed of article content loading is fast enough.

4. Efficiency3: It is easy to deal with subscription expired or e-coin run out.

5. Preference: You are preferred to use the system widely.

Figure 7: Three Payment Systems Usability Test Results

We choose to use these assessment criteria as they test key requirements of micro-payment systems as indicated in our earlier work on micro-payment system requirements development (Dai et al., 2001), and in other micro-payment work (Hwang et al., 2001).

In this study, ease of use, efficiency, and satisfaction/preference results mainly favored the client-side e-wallet NetPay system. However, this approach incurred an extra delay in page display due to communication from the vendor to the customer PC's e-wallet application, which the other systems don't have. Participants stated the article contents at different newspaper sites were easy to read without login and the balance can be checked any time. The server-side NetPay system allowed users to read articles on different computers, but customers needed to remember e-coin IDs and had to log into the new newspaper site when change vendor. The article content loading was very fast on subscription-based system, but the users found that it is not as convenient to change vendors. The users generally needed to spend more money in order to subscribe to the whole newspaper provided by each site. Open question results revealed that client-side NetPay was found to be significantly preferred over a subscription-based system. In addition, server-side NetPay was more preferred than subscription-based system for this e-tailing application domain.

Performance Evaluation

We ran two sets of performance evaluations, one on our original NetPay prototypes and a comparable macro-payment subscription-based system and one on modified versions of the NetPay prototypes, optimized to provide lower e-coin management overhead. This was done due to the large database overhead the original prototypes incurred for debiting e-coins.

The results of the first set of performance evaluations are shown in Table 1. The response time measures how long it takes for a page to be returned from the vendor site. The server CPU time measures the time spent in the vendor's server debiting NetPay e-coins.

Table 1: Initial Prototype Performance

System	Response Delay Time (average)	Server NetPay CPU Time Usage (average)
Subscription-based	16ms	-
Server-side NetPay	80ms	64ms
Client-side NetPay	950ms	-

From Table 1, the server-side NetPay takes 80ms-16ms=64ms for e-coin debiting per article and client-side takes 950ms-16ms=934ms total time, though the time to debit coins is taken by the client's e-wallet application, not the vendor's application server. The large overhead in the server for the server-side NetPay prototype is due to the database transactions it carries out to record coin updates and debits to redeem to the broker.

To reduce the e-coin debiting time, we created a transaction temporary file recording the data for redeeming instead of the redeeming database. Because of the application of such a temporary file, the e-coin debiting time decreases dramatically, especially for server-side NetPay system. The results are shown in Table 2. At the end of each day, the system redeems the coins or updates the database, and then deletes the records in the transaction temporary file. From Table 2, Server-side NetPay takes 30ms-16ms=14ms for e-coin debiting per article and Client-side takes 900ms-16ms=884ms after the application of the temporary file. The impact of the NetPay micro-payments on the vendor application server are greatly reduced, but the client-side e-wallet still incurs considerable response time delay due to the additional vendor->customer PC communication with it.

Table 2: Prototype Performance after using a Temporary File

System	Response Delay Time (average)	Server NetPay CPU Time Usage (average)
Subscription-based	16ms	-
Server-side NetPay	30ms	14ms
Client-side NetPay	900ms	-

Qualitative Analysis

With our qualitative assessment we wanted to measure the different characteristics of macro-payment and micro-payment approaches, and to analyse the differences between our two NetPay e-wallet models. The results of this assessment are summarized in Table 3.

Table 3: Qualitative Assessment Summary

Criteria	Macro-payment	Server-side Wallet	Client-side Wallet
Customer interactions	Subscribe (customer and credit card details) Login to Web site Article read Subscribe if move to another vendor +after subscribe/login simply read articles -must supply personal details -must subscribe for each vendor	Purchase e-coins from broker Login to Web site Article read Login to new vendor +after login simply read articles +only login to new vendor -must recall e-coin ID, password -must supply for each vendor	Download wallet software Purchase e-coins Article read +don't need login/password for any vendor +simply read articles -must download, install and have running client-side e-wallet software
Information retention	Need to remember username/ password. May avoid if use same PC always (can store in browser cookies)	Broker username and password needed to purchase coins. E-coin ID and password needed to access server-side wallet information	Broker username and password needed to purchase coins
Subscription cost	Low - if use moderate number of articles, more cost-effective for customer	N/A	N/A
Low (e.g., $10) High (e.g., $50)	High - need to use substantial amount of articles for benefit. No cost savings for customers if use multiple vendor sites
Article cost	N/A	Low - customer likely to read more. Vendor needs more read to cover costs	Low - customer likely to read more. Vendor needs more read to cover costs
Low (e.g., 2c) High (e.g., 10c)		High - customer likely to read less. Cost savings to customers Can price articles differently	High - customer likely to read less. Cost savings to customers Can price articles differently
Article requests by customers	Low - No cost benefit for vendor	Low - if low cost, vendor makes little profit	Low - if low cost, vendor makes little profit
Low (e.g., <10) High (e.g., >20)	High - Large number by many customers effects system performance	High - if high cost to customer, may be more costly than macro-payment approach High numbers impact overall vendor server performance.	High - if high cost to customer, may be more costly than macro-payment approach. Has large performance impact (in current implementation) High numbers impact overall response time of vendor server.
Vendor benefit	"Brand capture" of customers due to use of subscription to each vendor site.	If large enough vendor community can encourage movement, partnerships. Customers need to login to access wallets.	If large enough vendor community can encourage movement, partnerships. No login for wallet access needed but need wallet installed on customer PCs.
Vendor cost	Need to buy macro-payment supporting software and pay bank for facility. Need to price subscription to adequately cover costs.	Need to allow broker to take portion of overall customer payments OR broker takes costs from customer direct. Need to price articles so cost per article/ and number of articles used cover costs. The performance overhead on the vendor server is significant.	Need to allow broker to take portion of overall customer payments OR broker takes costs from customer direct. Need to price articles so cost per article/ and number of articles used cover costs. There is little performance overhead on vendor server but response time reduction for customer.
Broker cost	N/A	May charge vendors for each e-wallet request or portion of redeemed coin amount. May charge customer for each e-coin purchase. Possibility of high number of e-coin requests from vendors.	May charge vendors for each e-wallet request or portion of redeemed coin amount. May charge customer for each e-coin purchase. Low overall e-coin requests as client-side wallet brokers these.

Summary of Results

In summary, a macro-payment approach is more beneficial for the customer if they typically read a large portion of the online newspaper articles, or if a comparable micro-payment approach has a high-cost per article for the user. However, the micro-payment approach wins out when the customer typically users a small portion of the articles, articles are low-priced and if the customer reads articles from multiple newspapers and can use their e-coins across any of these vendors. There is a performance cost for the vendor in providing a micro-payment approach in terms of time taken to track e-coin spends and redemption. However, there is also normally a high cost to the vendor of providing macro-payment support for subscription purchase.

One interesting issue is whether vendors would "buy-in" to a micro-payment system approach. By using subscription-based macro-payments to access information, vendors can lock in customers—i.e., achieve "brand capture" and discourage customers from moving to other information sources, e.g., other e-newspapers, as they have already made a significant financial commitment to one newspaper. Similarly, there needs to be sufficient vendors sharing the same micro-payment system and "currency" to allow useful movement by customers from vendor to vendor. In addition, the software maintenance overhead of installing a micro-payment system must be considered by vendors. One approach is to adopt a portal-based approach to accessing multiple vendors through a single micro-payment enabled portal, which does the debiting and redeeming of spending on behalf of multiple vendors.

Other alternatives do exist to providing macro- and micro-payment models as described in this chapter for e-tailing systems. The most common is the provision of services to users for free but with the use of advertising embedded within pages or as pop-up windows. However, studies have suggested that vendors would prefer a payment mechanism that is on a per-usage basis, either subscription-based or micro-payment-based to provide greater reliability of income. A key outstanding challenge with micro-payment systems is being able to spend currency (e-coins) at a wide range of vendors—if the customer must purchase different "currencies" for different groups of vendors then this will be both inefficient in terms of expenditure and incur overheads of the customer memorizing different usernames, e-coin IDs, and passwords. One approach is to support inter-broker micro-payment e-coin exchange transparently when the customer visits a vendor that uses a different broker's currency.

---