Introduction

In Chapter 4, "System Performance Monitoring," we talked about monitoring the performance aspects of a system as a way to understand some of the characteristics of a workload. We also looked at some tools for monitoring individual applications. These tools typically provide information about the system's performance at a specific point in time. Some tools also monitor the system at regular intervals and log performance statistics, allowing a performance analyst to watch how the system's performance changes over time. Although performance monitoring data can be critical to understanding how a workload operates over time, such data is only a one-dimensional view of overall workload response time. For instance, very few workloads run at a steady state of operation for their entire lifetime. Workloads are primarily responsive to human interaction, although there may be other significant factors that generate workload activity. Because these interactions are so random, periodic monitoring often happens with little insight into the types of requests the users are generating or how representative those requests are of the general load that the workload can handle. Further, although this style of monitoring is useful for identifying performance bottlenecks, it is less effective for understanding how well the workload could or should perform. It is also difficult to evaluate the potential advantages of changes in hardware or configuration or predict what will happen when the system is reconfigured.

In Chapter 4, we mentioned that statistics gathered by most monitoring tools are easiest to understand if they can be put in context based on past workload behaviors. For instance, when a workload has been working well for months, and suddenly the response time increases by an order of magnitude, being able to compare current monitoring data with past monitoring data can help provide insights into the problem area. This technique is useful for problem analysis; however, general system tuning is much simpler with a predictable workload. Also, a workload that depends on tens or hundreds of users generating interactions with the system is difficult to set up and difficult to make repeatable. Again, having a predictable workload that can be automated and that can simulate the interactions of the users allows a performance analyst to run the workload repeatedly, changing selected aspects of the environment and observing the changes in the underlying system statistics.

Repeatability and consistency are the primary indicators of a potential benchmark. With those two factors, you can perform a statistically valid scientific application of a control environment and can analyze the results of changes to the control environment. Using the monitoring methods discussed in this chapter, you can identify system statistics that are constant from run to run, those that change slightly, and those that have dramatic change as various aspects of the configuration are modified. These basic tools allow you to perform "what-if" scenarios for analysis in a basic sense. Exploring these scenarios allows you to tune your workload and your environment to get the best responses out of your system and to decide when portions of your system need to be improved. In some cases, you can also identify bottlenecks within the applications using this process, leading to direct modifications to the applications supporting your workload.

Further enhancements to this benchmark would include the capability to measure those factors of your workload that are critical to its overall throughput. For instance, on a file server, the capability to measure the number of files opened and sent to the end user over a period of time may provide a key measure representing the overall throughput of your file server. A web server may choose to measure the number of network connections created as a useful measure of work being done. Being able to identify a few key measures for a consistent, repeatable workload allows a performance analyst to use simple metrics to compare the results of various changes to the system environment. And this is exactly the point of benchmarking: creating a frame of reference for evaluating and improving the overall performance of your workload in your environment.