Facing the Facts--January to March 1999

Facing the Facts—January to March 1999

The second user test took place during the first 2 months of 1999. During this round of evaluation we made fewer compromises. The number of subjects was increased to eight. We were also careful to select subjects that better represented the intended user group. Yet some compromises had to be made. We were forced to use internal subjects, namely, employees of Nokia, since the one-row concept was regarded as highly confidential. Also, our prototype was still not handheld, so we were not able to test actual mobile use. However, we proved the value of early testing and decided to do it again. And again the testing turned out to be very worthwhile.

Eight subjects used the one-row keyboard for about one week. They were allowed to type in a free style, using as many fingers as they wanted. Since the keyboard prototype was now pretty small, all the users ended up using some variation of the four-finger strategy, as we expected. All subjects were regular phone keypad users. Earlier they had participated in a similar evaluation with T9, though none was a regular T9 user. That test was now used as a benchmark.

As users were familiar with the phone keypad, if not with one-row keyboard, we expected them to be pretty slow in the beginning. We measured their typing speed throughout the test, and expected it to surpass T9 speeds after some time. This hypothesis was based on the fact that regular T9 is operated mainly with one hand, whereas one-row is operated with two-handed input. We surmised that two hands would turn out to be more efficient in the long run. The hypothesis turned out to be false.

Figure 7.5 shows the typing speed learning curves of T9 and the one-row keyboard. One-row started from a lower level, as expected, but contrary to expectations the curves never crossed. In fact, they seem to diverge a bit. This finding was supported by subjective findings. Several users complained that the one-row keyboard was slow and tedious to use.

Figure 7.5: Typing speed during the evaluation with regular T9 and with the one-row keyboard.

After careful analysis of the results, there was only one possible conclusion: the difference is real. All subjects were consistently faster with regular T9 than with one-row, and there were no exceptions. Subjects’ spontaneous comments about one-row were clearly critical. The team was stunned by the result; the two keyboards were based on the same operating principle. As one of them was used mostly with a thumb, the one with four-finger options was supposed to be an improved version. How could we have such a result?

For a while team members still tried to challenge the result of the user test: “C’mon, we ran the tests with only eight subjects. . . .” Some even took the trouble to learn to use the keyboard themselves, but were finally driven to the same conclusion. The keyboard was neither easy nor fast, and we had no choice but to face the facts.