Efficiency of the Trials

Field trials achieved a semimobile usage context, which was a much more realistic situation than in laboratory studies. The users alone were responsible for activating services, whereas laboratory studies guide the subject through the process, and there are fewer opportunities for user-initiated decisions concerning use.

The business case for wireless multimedia services was less convincing for operators during MOMENTS than it is currently. Doubts about investing in mobile multimedia could have affected both motivation and resource financing as the project advanced into more demanding phases. Under these circumstances, the field trials gave the operators a chance to get deeply involved in testing and let them develop some competence with the technology, the establishment, administration, and analysis of the trials.

We needed to judge the costs of the project for each partner. The efforts in person-months were calculated over 3 years after the actual project and follow-up reports in order to determine project efficiency. These numbers included all phases from the initial organizing of trials, through the fieldwork and time spent evaluating study results. The division of effort in the service trial project phases was 35 percent for the organization of field trials, 33 percent for the operation of field trials, and 32 percent for the evaluation. This was a good result; if the investment in organizing and running trials is inadequate, the validity of results decreases because of insufficiency of data. Increasing the evaluation efforts later on cannot improve results where usage activity has been too slight.

In large trial projects it's hard to estimate costs and efforts needed in order to reach all original objectives. In this project the funding was shared between partners, and in fact two new partners were invited into the project at the beginning of the second project year. The lesson learned for similar projects is that flexibility in replanning is essential for the entire project duration.