Interpreting Test Results

Many parts of the usability test require the tester to fathom what's going on in test subjects' heads when they're not saying anything ”and that can be a tricky business. I've seen people who struggled with an application and, when presented with three choices, said the two "wrong" choices several times. When I first witnessed this, I was amazingly frustrated and wanted to blurt out "Just say the other choice!" But professionalism prevailed and eventually they said the right choice (after a LONG while).

At the time I thought that either the test subjects must be stupid, or I was totally unclear when I explained the task. Actually, I was wrong on both counts. What had happened was that the subjects had heard part of an introduction prompt that caused them to form a different mental model than the one the designer had in mind. The test subjects tried to adhere to this mental model until finally giving up and trying the "correct" command. It was at this point that they began to form the correct mental model ”but clearly in a less than optimal way. When I realized what had happened, I suggested that the designer change the introductory message ”and the problem went away.

On another occasion, I saw a test subject struggle with an application ”only to later write on the post-test questionnaire that she loved it! I was flabbergasted! The explanation? The test subject hadn't remembered struggling with the system, because she eventually got her tasks accomplished. But she did remember ”and appreciated ”the way the system helped her when she was "stuck." This example teaches us an invaluable lesson: The perceived quality of a system isn't always related to the subjects' ability to fly through it without problems ”but rather, how well it helps users when they experience a problem.

As these experiences suggest, usability tests are vital ”but they are only one step in the process from development to deployment. That's why we continue to observe and change the application during the pilot test.