Recipe 9.10. Developing Test Procedures for Your Site | Web Site Cookbook: Solutions & Examples for Building and Administering Your Web Site (Cookbooks (OReilly))

Problem

You need to test your site to make sure it meets the goals you set for it, serves its visitors, and performs optimally.

Solution

Understand the types of testing that need to be done and incorporate them into your site building and maintenance process. The four main types of web site testing are:

Usability testing
Browser testing
Quality assurance
Load testing

Discussion

A former coworker and veteran web site tester once summed up her job to me: "Don't let anything get on the site that makes us look stupid." Of course, the reasons to test go beyond this simple mantra, but it also encapsulates (in its own blunt, self-serving way) why testing your site early and often matters. After the many hours you and others put into a site from concept to launch, failure to verify your plans and assumptions along the way can reduce your efforts to nothing more than a fool's errand.

Usability testing

Testing begins with your prospective visitors. Although you (like my old colleague) may worry about how an untested or unproven site will reflect poorly on your company or organization, the ones left looking (and feeling) stupid often are the site's naive users. When web surfers encounter a problem or confusing page on a site, they tend to blame themselves for what's gone wrong. The stubborn ones may try again, but many will give up and leave the site. If you haven't done the work to flush out and fix problems users may encounter on your site, you won't have to worry about the site continually making you look stupidafter a while, no one will be looking anymore.

The sooner in the site-development process you start collecting and acting on feedback from typical site visitors, the better your site will be. Fortunately, you don't have to interview hundreds of potential visitors or spend a fortune to accomplish meaningful usability testing. As Jakob Nielsen and other well-known usability experts have pointed out, any amount of usability testing is better than none, and a significant number of potential problems can be discovered by doing testing with just a handful of people.

Usability testing is not a means by which to justify a site design or reject a controversial color scheme. Rather, it is a means of soliciting feedback from potential users who are far enough removed from the creation process to provide a different and meaningful perspective on how it will "play" in real life. Likewise, usability tests are not focus groups, and from the subject's point of view should not even be considered a "test"lest they fear getting something wrong. Usability testing should be done one-on-one in a comfortable and non-threatening setting for the subject.

If you have the foresight to do some testing before coding begins, you can use low-tech paper prototypes to represent your site. These can be nothing more than rough, hand-drawn representations of how you intend the site to look and how users will interact with it. Simple prototypes often are the best way to gauge how well users understand your site's concept and organization. When testing with an actual working copy of the site, interview your test users in their own homes or offices, using their own PCs. Here, you want to find out if people can actually use the site, how long it takes them, and if they understand your navigation, forms, buttons, and icons. Give them enough guidance to get the feedback you want, but refrain from leading them to the answers.

Browser testing

With methods in place to incorporate the results of user testing into your web site operations, you should turn your attention to your code and how it gets rendered by various browsers. Here again, early testing is crucial. The time to discover display inconsistencies among various browsers is not after every page on your site has been coded, but when the first mockups or templates have been translated to HTML. If you're coding to web standards, then the first step in browser testing is code validation against the document type, or DOCTYPE, you're using for your pages. (For more information, see Recipe 4.1). Nothing beats visual confirmation, though, so you also should take the time to install and view your sample pages in the browsers you expect your visitors to be using. (For more information, see Recipe 2.1 and the See Also section of this Recipe for an excellent online archive of old browsers).

Quality assurance

Quality assurance (QA) is an integral part of site maintenance and updating, as it combines the ability to think like a typical user with the site builder's acumen for understanding why something has gone wrong. The QA process incorporates the copyediting functions that every publisher needschecking pages for spelling, grammar errors, and style inconsistencieswith the technical proofreading skills unique to the web: checking pages for broken links and unloaded images, verifying error messages, and provoking a web site into revealing its Achilles' heel. Ideally, every page on your site should go through QA before it is published. But your own role as producer, designer, or contributor to a site hinders your ability to QA your own work. Where possible, recruit or employ others to do this testing phase for you or consider one of the many online tools and utilities that will perform this service for you. Many of your visitors also will let you know when they find a problem. For more information on getting the most out of this dialog, see Recipe 9.3

Load testing

Performance benchmarking based on load testing can help you gauge how well your web server hardware and configuration can meet the needs of your site's traffic as it grows. It also can provide you with a useful metric on the effects of site enhancements you undertake, such as code and image optimization or caching.

But like any statistic, a measure of web site performance is most useful when you have previous statistics for comparison. That's why you should begin doing load tests on your site early in its lifetimenot when you begin to suspect a problem with performanceand continue the tests on a regular and consistent basis. In other words, a load test performed at the same time every week or month will have better validity in exposing performance trends than load tests done irregularly.

In addition to the many online load-testing tools (such as http://www.netmechanic.com) and paid monitoring services, Apache installs its own command-line benchmarking tool called ab. In order to ensure that the use of ab does not affect the results of your test, you should run the utility on a separate machine from the one being tested. From a shell prompt on a separate machine from your web site, run this command:

 ab -n 100 -c 5 http://yoursite.com/index.php

The -n and -c flags are used to instruct ab to request the site's home page (http://yoursite.com/index.php) 100 times in batches of five concurrent requests. Although the output from ab (sample below) may not mean much on its own, a series of analogous reports collected over time can help you stay on top of your site's performance and anticipate the need to perform tune-ups or upgrade hardware before slow server responsiveness drives visitors away.

 Benchmarking yoursite.com (be patient).....done Server Software:        Apache/1.3.33 Server Hostname:        yoursite.com Server Port:            80 Document Path:          /index.php Document Length:        13642 bytes Concurrency Level:      5 Time taken for tests:   17.089 seconds Complete requests:      100 Failed requests:        0 Broken pipe errors:     0 Total transferred:      1406795 bytes HTML transferred:       1389965 bytes Requests per second:    5.85 [#/sec] (mean) Time per request:       854.45 [ms] (mean) Time per request:       170.89 [ms] (mean, across all concurrent requests) Transfer rate:          82.32 [Kbytes/sec] received Connnection Times (ms)               min  mean[+/-sd] median  max Connect:       48    53    3.2     53   65 Processing:   399   784  172.8    753 1296 Waiting:      347   783  172.8    752 1295 Total:        399   837  173.5    806 1349 Percentage of the requests served within a certain time (ms)   50%    806   66%    891   75%    927   80%    965   90%   1049   95%   1200   98%   1327   99%   1349  100%   1349 (last request)