Chapter 8. Selecting the Right Test Tools

Selecting the right test tools is critical to the success of your performance test. You depend on performance tools to capture reliable and repeatable data during testing. However, not all testing tools prove suitable for all testing situations. In fact, we occasionally encounter obscure tools that actually generate performance problems! In this chapter, we discuss selection criteria for performance test tools. As we discussed in Chapter 7, most performance tools come with a recording tool and often their own scripting language. Therefore, selecting a tool implies an investment not only in the tool software, but also in the test scripts you generate using the tool. Selecting a good, robust tool saves money in replacement costs later, and also in the time required to build your test script library.

As we'll discuss in Chapter 11, successful tests depend on two critical elements:

• Simulating the anticipated production environment

• Obtaining reliable and repeatable measurements

You need a test tool to support these success criteria. Unless your tests adequately simulate the anticipated production environment, problems resulting in significant production issues often go undetected. Likewise, reliable and repeatable measurements allow you to correlate tuning changes with improvements rather than measurement error.

Test tools continue to rapidly improve their capabilities. For this reason, we do not include a feature-to-feature comparison between different tools. Instead, we provide a requirements framework for assessing performance tools, and discuss some of the key feature differentiators. Appendix C includes a checklist summarizing these requirements.

In general, the Internet and the market place both contain plenty of low-end, free, or shareware load drivers, as well as high-end load drivers. Some of the low-end drivers include tools such as Apache Bench, Apache Flood, Jmeter, or Microsoft Web Capacity Analysis Tool (WCAT). These tools typically work best to test static web sites or simple dynamic sites. High-end load drivers come from the leading automated software quality tool vendors , including Mercury Interactive, Rational, Segue, Compuware, Empirix, and RadView. ^[1] Tools at this end of the market generally work best for large or complex web sites.

^[1] See Richard V. Heiman, "The Distributed Automated Software Quality Tools Market Forecast and Analysis, 2001 “2005," IDC Bulletin 25176 “July 2001. Retrieved December 27, 2001, from the World Wide Web: <http://www-heva.mercuryinteractive.com/company/pr/idcreport/25176.htm>. Table 1 lists Mercury Interactive Corp, Rational, Segue Software, Compuware Corp, Empirix, and RadView Software as the top six vendors of distributed automated software quality tools by 2000 revenue.

Appendix C includes a list of vendors of performance testing or monitoring products. Of course, this is only a partial list and does not contain every tool available. Also, we do not endorse, recommend, or guarantee the tools listed, but merely provide these names for your convenience.