Just-in-Time Usability Engineering

Today's technology may change too rapidly for traditional usability engineering. The designer can't identify users and their tasks before design, because the technology being designed will create its own tasks. In many cases, the application is strongly defined by standards that showcase the technology, leaving the designer to imagine what it might be good for.

Specialist application teams and just-in-time usability allow Nokia to produce usable systems even under these constraints. And although Japanese factory management techniques were not the inspiration for these design processes, the similarities are striking. The clearest parallel is to just-in-time supply requests in the factory. In the same way, Nokia's usability experts must respond quickly to requests for information or evaluation. There is seldom a backlog of usability studies to draw on, because most situations were never planned or imagined.

A designer calls the team's usability expert: 'The wireless carrier's messaging standard has just changed,' he says. 'We can't send as many characters as we thought, but we can still receive them . . . is there a usability problem?'

Or, 'We're using a nonstandard display on this product, and the mechanical design can't be changed . . . now we need a readable font.'

Or, 'We've designed this new browser add-in to meet the common standard . . . can you help us figure out what users might do with it?'

This is not the traditional, carefully planned, cradle-to-grave iterative design approach. This is reactive engineering, where the rational assumptions may be toppled at any moment like a line of dominos. And to allocate an army of researchers to cover the range of possibilities with prospective studies would be wasteful, indeed.

To handle these sudden, unplanned requests, the usability experts-I'll call them the 'platform usability group' to distinguish them from UI researchers-must be generalists. Like the Japanese factory workers, they must be ready to pick up any task, with help from their colleagues and research groups in sites scattered across the world, and provide an answer, or even a firm 'don't know,' at the shortest notice.

'This is the new hardware proposal . . . we have to make a go/no-go decision now. Are the keys OK . . . or does it need testing?'

Or, 'The wireless operator wants this new message during roaming calls. It has to be in the software release for next week. Are there usability issues here?'

The hardware proposal is an example that exemplifies the just-in-time approach. The industrial design for the product had been completed and evaluated months earlier, but changes in the predicted market and the schedules of other products demanded a sudden redesign in a smaller form factor. The redesign was completed in record time. Coincidentally, the new wax model was presented to management a few days after the first working prototypes of another product were distributed to internal users. Users had begun to report that the key layout on the working prototype encouraged a fairly serious type of error, and a redesign was being considered. Managers now were worried that the design represented by the wax model, which had a key layout similar to that of the prototype, would produce similar usability problems.

Usability testing could shed light on the question, and every usability expert has the expertise to run the tests. But there was no way to test for key press errors on the wax model, and a decision had to be made immediately. Usability experts turned to their knowledge of human-factors techniques and performed link-analyses, evaluating the hardware with reference to usage patterns of keys in the standard Nokia UI. They concluded that even if errors did occur, the risk associated with them was low-the user could easily recover. The industrial design group was able to make minor changes to reduce the chance of error even further, and the design was approved.

It is no easy task to give immediate evaluations of products that stretch the boundaries of technology. The educational backgrounds of the platform usability group reflect this broad challenge. Most have at least a master's degree; several have additional education or PhDs. In addition to the obvious disciplines of human-computer interaction and various behavioral sciences, the group includes students of economics, anthropology, and computer science. Some are programmers, some have studied statistical techniques, and some have managerial strengths. Several are trained in hardware ergonomics. All must be familiar with Nokia's UI styles and basic features. All interact regularly over the Net (Internet) and in person, despite their global distances and time differences.

A usability engineer in Nokia's San Diego site comes in at 5 a.m. several days a week, to join email and phone conversations with colleagues in Europe. 'There's also less traffic on the freeway,' he notes.

Responsibility for quality is a third point where the Japanese and the Nokia approaches agree. Just as factory teams are expected to maintain defect-free quality in their output, each Nokia application team is ultimately responsible for the usability and overall quality of its own work (an approach pioneered by Gene Lynch and his colleagues at Tektronix.^[17] The key team members are the UI designer and the software engineer. There is a localization specialist to define texts, a platform usability expert, and test engineers, and there may be graphics and sound support as well. For complicated applications, the user guide author may even join the team.

The team's multifaceted background makes it ideally suited to perform various forms of structured usability inspections, originally developed through research into user interface design processes in the 1990s.^[18] Nielsen's formalized version of the traditional 'heuristic analysis' is probably the best known of these. Heuristic analysis, performed in many fields, involves examining a system for violations of clearly stated guidelines, or heuristics. For UI engineering, Nielsen suggests a small set of widely applicable heuristics, such as 'Prevent errors.' His research also shows the value of combining comments from several inspectors. The analysis can be especially productive when the inspectors have 'double' expertise, covering usability as well as the system domain area.

Walkthroughs are another inspection method arising out of existing practices, in this case the code walkthroughs of software engineers. The 'cognitive walkthrough' is designed to apply theoretical work by Clayton Lewis and Peter Polson. Each walkthrough involves a step-by-step examination of the sequence of mouse clicks or similar actions required to complete a typical user task with the system under design. The inspection process uses specific questions and criteria to help the inspectors predict the mental behavior and likely errors of a novice user.^[19] Randolf Bias' 'pluralistic walkthroughs' are another approach, placing less emphasis on cognitive theory and more on the social dynamics of bringing critical stakeholders together in the face-to-face inspection meeting.^[20]

The inspections are done by experts, without involving real users. This allows them to be completed well before code or even simulations are written. Combined with on-line inspection of design documents by other teams and usability experts, they form a first line of defense against egregious usability errors. The inspections also help focus subsequent user testing on critical issues. Often these involve texts or icons in the display. If the inspections identify the problem areas, alternative textual or graphical items can be tested using surveys. This provides a wider coverage of markets and user groups than does task-focused testing in a usability lab. Sometimes the situation is more complex, and the user's behavior depends on a mixture of background knowledge and problem-solving techniques that is difficult to predict without observing real people. In these cases, more traditional usability lab methods can be used.

Reviewers of an application that is patterned on its PC counterpart note that one screen has an unusual mixture of user data and default functionality. It seems to be what's needed, but will the user be confused? The inspection raises the question; we'll consider user testing as a route to the answer.

The usability expert joins the inspections and may arrange testing, but never acts as a final authority. In yet another concordance with the Japanese model, the work is a team effort. The UI designer and the software engineer are typically the final links in the decision chain, but this reflects operational necessity as much as formal authority. Usability issues can be discussed by any member of the team, and the final design is an informal consensus.

A software engineer resolves a data-calls usability issue by applying the 'mom' test: 'If my mom wouldn't know what to do with this menu, we don't need it.' The team decides that she is right.

The involvement of the entire team, especially the interaction designer, provides a baseline of usability that makes it unnecessary to usability test and iterate every element of each new product-an impossible task with a product line on the scale of Nokia's. And even though plans and resources are allocated for testing major new designs at several stages, the focus of the testing is frequently revised to address controversies that have just surfaced.

A critical series of screens in a new UI style is designed to use icons as control elements. The aesthetic improvement over a purely textual interface is evident, but there are lingering questions about the usability of the icons. An already scheduled usability test is replanned to focus explicitly on the issue.

The text-versus-icons study is a good example of the just-in-time approach. This question had come up earlier, and was even tested as part of an exploratory effort considering several new designs. The chosen design incorporated many novel features, and it was expected that several rounds of design, testing, and redesign would be required. Progress was being made, but after two rounds of testing, users still had problems in coming up to speed in the new system. The issue of the icons was raised again.

Another round of testing in several countries was already scheduled, but the test plan was modified in the days just before the test. Instead of exposing users to the latest design and watching for problem spots through the usual think-aloud protocol, two alternative designs were tested: one with icons and one using texts instead. Task times and errors were measured and analyzed statistically. The test results alone did not answer the question, but they tipped the balance to allow a decision that all parties could support without further discussion: The text-based design was selected for further development, and the iconic system was postponed until further research could make it acceptably usable.

Opportunistic actions such as revising existing test plans 'on the fly' are another defining feature of just-in-time usability. The opportunistic approach is reminiscent of the opportunistic planning described by Barbara Hayes-Roth and Frederick Hayes-Roth in artificial intelligence (AI) literature.^[21] It allows the small team of platform usability experts to make the most effective use of their time, rather than following a slavish routine of usability evaluation and testing for every design, no matter how trivial. It also allows the group to emphasize another goal that parallels the Japanese approach: continuous improvement.

A strictly planned route to continuous improvement would entail a lifetime cycle of design, implementation, and tracking of problems in the field. Some of this is possible, but it could take years to feed changes back into the next version of a product-and by that time, much of the technology may be obsolete. The opportunistic approach and Nokia's platform support for product development allow this cycle to be compressed, by feeding information from several overlapping product cycles into the development effort.

An implementation of instant messaging on a limited-market prototype is studied with internal users. The technology in this product is already outdated, but studying it raises flags that will help avoid problems in the revised system now under development.

The 'out of the box' experience for a new product is tracked with diary studies for several months, to identify usage patterns and major difficulties. No significant changes will be made to this product on the basis of the study, but the results feed into our general understanding of design. (One finding is that mobile phone users do read their user guides when faced with a difficult task. This will help designers decide how to distribute information among the user guide, on-screen help, and other resources.)

This example points to another situation where the usability group reacted to a fortunate set of circumstances. The project was a joint effort between Nokia's usability experts and a master's student doing thesis work in a local university, allowing the usability team to increase their effective resource count temporarily exactly when it was needed. Interestingly, part of the study was patterned after similar work that had been performed with personal computer users almost a decade earlier-a study that had been the dissertation project of one of Nokia's usability experts.

^[17]S. J. Grossman, G. Lynch, and M. Stempski, 'Team Approach Improves User Interface for Instruments,' Electronic Design News, June 4, 1992, pp. 129-134.

^[18]J. Nielsen and R. L. Mack, eds., Usability Inspection Methods. New York: Wiley, 1994.

^[19]C. Wharton, J. Rieman, C. Lewis, and P. Polson, 'The Cognitive Walkthrough Method: A Practitioner's Guide,' in Usability Inspection Methods, J. Nielsen and R. L. Mack, eds. New York: Wiley, 1994.

^[20]R. G. Bias, 'The Pluralistic Usability Walkthrough: Coordinated Empathies,' in Usability Inspection Methods, J. Nielsen and R. L. Mack, eds. New York: Wiley, 1984, pp. 63-76.

^[21]B. Hayes-Roth and F. Hayes-Roth, 'A Cognitive Model of Planning,' Cognitive Sci. 3(21): 275-310 (1979).