Message from Rome

Saturday afternoon I received a text message from Rome: “Lots of errors in the prototype and test material. Need to be fixed.” Apparently the part of our test team that had flown to Rome to verify and check the test arrangements had done its job.

Once in the field, the usability test team was extended to include a native moderator, an observer, and an Italian-English interpreter. Saturday had been a training day. The vanguard of the test team trained the local staff to perform usability testing according to a prepared script, taught them our test objectives and got them familiar with our paper-based usability tools and made sure that everybody knew their role and responsibilities. During the training day it became evident—thanks to the local staff—that our terminology and test tasks were merely translated, not properly internationalized. Because of this some screenshots looked “funny” and some test tasks were described in an obscure and unnatural way. A phone conference was arranged, and the necessary changes were agreed on and implemented overnight.

On Sunday, the vanguard spent the whole day at the Formula 1 race at San Marino observing Italian mobile phone culture. The rest of the test team headed off to Rome from Finland.

We repeated this training-day schedule in each country of the test program, and it worked well for us. Apart from its project objectives, the training day also taught us details about language and local communication patterns. Test tasks were fitted to local culture, and localized test material got its final verification. A lot of language mistakes were corrected beforehand, and the suitability of the tasks for the local culture was improved. For example, in Italy the daily use of voice mailbox—casella vocale—service was very different from the use cases anticipated, and we had to scramble to make modifications that would reflect local use. Regional facilitators thus assumed a secondary role as domain experts in the test preparations. It’s not enough, we learned, for usability evaluators to be able to converse in a given language; they should also be familiar with the idiosyncrasies of the foreign language.^[1] In a way, we ended by testing the user interface twice: first, by walking through it with local staff and again in administering the formal tests.

^[1]A. Yeo, “Usability Evaluation in Malaysia: An Exploratory Study of Verbal Data,” in Proceedings of 4th Asia Pacific Computer Human Interaction Conference: APCHI 2000, Singapore, Nov. 27–Dec. 1. Amsterdam: Elsevier, 2000.