User Testing

Any process based on observation must take the back seat to acts of creation. Programmers create. The usability discipline tacitly hands the reins to the programmers, saying, "You build it, and then I'll test to see how well you have done." But in this fast-moving, high-tech world, after it is built, it ships. Post-facto testing cannot have much influence on the product.

To me, usability methods seem like sandpaper. If you are making a chair, the sandpaper can make it smoother. If you are making a table, the sandpaper can also make it smoother. But no amount of sanding will turn a table into a chair. Yet I see thousands of well-intentioned people diligently sanding away at their tables with usability methods, trying to make chairs.

User Testing Before Programming

It is certainly possible to perform user testing before programming begins, but the nature and value of the process changes dramatically. This kind of testing is similar to the pure research that one would expect to find in a university setting. A colleague at a major software company performed a classic user test that simultaneously demonstrates the strength and weakness of this pre-facto user testing. He wanted to determine the effectiveness of the status bar at the bottom of the screen. He had people use a spreadsheet program to perform some innocuous task, and about every five minutes a message would flash across the status bar saying, "There is a $50 bill taped to the bottom of your chair. Take it!" In a full day of testing with more than a dozen people, nobody claimed the cash.

The insight that users don't pay much attention to what is displayed on the popular-among-programmers status bar is certainly valuable. It doesn't shed much light on the underlying problems, though: What constitutes "status" worth displaying? Should it be displayed at all? Where should it be displayed? Those design problems remain as open as they ever were.

Fitting Usability Testing into the Process

The professional literature is filled with detailed advice on how to perform tests, but it says little about inventing something to test if the product doesn't already exist. In practice, some simulacrum must be created and tested. These generally take the form of either a quickly written prototype program or a "puppet-show" made from paper cutouts or some equivalent, low-tech material.

You can learn a lot about users' reactions from paper puppet-shows, but what gets tested can still be quite inappropriate unless design is done first. Also, the personal presence of the tester inevitably looms large in this form of test, and a word, nod, or glance can easily skew the test's results.

For the most meaningful results, you have to do prohibitively expensive comparison testing by creating two programs to test against each other. Even then, all you learn is that one of the candidates is better than the other. You don't know what is the best you can achieve.

Thoughtful user testing can uncover a designer's incorrect assumptions. Exposing your design work to users and then redesigning iteratively is always better than not doing so. Some new technologies, such as voice recognition, are so untried that the insights provided by basic user testing can be of great value.

Arguably, the most valuable contribution of usability testing is made when programmers are forced to sit behind the one-way mirrors to view typical users struggling with their programs. The programmers are shocked and incredulous, shouting sentiments like, "You are testing mental retards!" Usability testing is a useful whack on the side of the head for recalcitrant software engineers, showing them that there is indeed a problem. It can serve the same purpose for management, too.

To paraphrase the toothpaste people, user testing has been shown to be an effective, decay-preventive technique when used in a conscientiously applied program of Goal-Directed design and regular professional care. The key here is to remember that other factors can have an even greater effect.