Buy versus Build

In the early years of web site testing, many organizations often resorted to writing their own test tools. However, a wide variety of excellent free and licensed performance test tools exist today. If the free tools do not meet your needs, we strongly suggest purchasing the level of performance test tool support you require rather than writing your own. If you can't afford these tools, consider leveraging some of the low-cost or free tools with some additional custom code.

However, as your site grows and testing requirements become more sophisticated, you may quickly need the capabilities of a high-end load driver. Your test scripts, automation, and customization represent a significant investment, and switching test tools comes with a much higher price tag than just the cost of the software. Pick a tool that meets your needs for awhile.

In the monitoring space, you need the capabilities that come with the operating system and applications to gather data. Your choice is whether to gather this data manually, through your load driver, or with custom tooling. For simple tests, using the free system and application tools to gather metrics and manually watching and recording the monitoring data typically suffices. If you have many systems under tests or long-running tests, look for a load driver that captures this data across the set of servers, or develop custom tools (usually scripts) to collect and correlate the data.

If you struggle with performance problems in a complex environment, consider investing in specialized monitoring tools, especially for application servers and databases. Be sure to invest in training to make the most of these tools. In general, purchasing test tools and investing programming time and resources in your business applications is the right trade-off. However, if you run a lot of tests, consider applying some resource to test automation to simplify execution and analysis.

In the end, select a performance test tool for your budget, but be prepared for a higher price tag than expected, especially if you need advanced runtime or reporting features. Spending in the neighborhood of $100,000 on testing tools seems excessive, but if your production site does not perform under load conditions, the loss of customers and revenue quickly justifies the expense of adequate testing.