CPU Utilization

The CPU provides the processing capability for the various machines involved in your testing. Measuring CPU utilization throughout your system gives you a tremendous amount of information about the effectiveness of your test and any tuning adjustments you've made. CPU utilization measures how much of the CPU (or CPUs) available are actually engaged in work. A fully engaged CPU (a CPU running at 100% busy) cannot process more work, so giving the CPU additional work does not result in increased throughput. Likewise, a CPU running at low utilization indicates a system not working as hard as it might. (See Chapter 15 for more details on correlating CPU utilization to system performance.)

During your testing, capture CPU utilization on all systems involved in the testing. This includes test clients , HTTP servers, Java application servers, and database machines. Don't forget any mid- tier systems or host systems involved in the test as well. CPU utilization is a cheap thing to measure; it gives you a lot of information about the remaining capacity of the equipment involved in the testing. Also, comparing CPU usage among the systems involved in the test gives you insight into potential bottlenecks.

Monitoring CPU on UNIX Systems

UNIX systems provide some simple tools for measuring the different components of CPU utilization along with some other metrics. The most familiar UNIX-based monitoring tool is vmstat . To run vmstat , simply type in vmstat followed by the refresh interval (in seconds). For example, typing

 vmstat 5 

tells the system to sample the system every five seconds. The resulting output is shown in Figure 12.1.

Figure 12.1. Sample vmstat output

Notice the last three columns , where vmstat breaks the CPU down into multiple components: user ( us ), system ( sy ), and idle ( id ). ^[1] In rough terms, user CPU indicates the amount of CPU utilized by an application running on the server. System CPU indicates the CPU used by lower-level processes. For example, excessive HTTP listeners sometimes push up the system component of CPU. Also, all Java virtual machines we've seen tend to drive up the system component of CPU slightly (usually under 15% “20%) while the application executes. Add the user and system components to determine total CPU utilization.

^[1] A definition of each field shown in Figure 12.1 is found at <http://www.uwsg.iu.edu/usail/man/linux/vmstat.8.html> retrieved from the World Wide Web March 15, 2002.

The idle component indicates how much of the CPU capacity is going unused. Idle indicates the machine capacity still available to handle processing. In this Linux vmstat , CPU wait time is not explicitly broken out. Here, wait is detected by using the process stats on the left, indicating running ( r ), blocked ( b ), and waiting ( w ). Nonzero wait values indicate the CPU wants to work but is waiting for some unavailable resource. As discussed in Chapter 13, high wait usually appears when the system experiences excessive disk I/O. Also, we sometimes see this condition during network or remote system failures. Some systems provide an iowait component of CPU time in their vmstat reports . This also gives you an indication of how much time the CPU spends waiting for unavailable resources.

The vmstat report also provides information on paging and disk I/O for the system. These statistics sometimes provide insight into excessive paging or I/O problems within the system. Watching the cs column monitors context switching, and monitoring the swpd column shows system swapping statistics. As we've mentioned before, if you find your JVM paging, reduce the JVM's heapsize or add more memory to your machine. A paging JVM does not give acceptable performance.

Vmstat sends results to stdout by default. To capture the results to a file, redirect them via the command line. We strongly recommend capturing this data to a file for future inspection. We also suggest developing a naming convention to coordinate this information with reports from other systems. Usually a naming convention incorporating the name of the machine, the date, and a test run identifier allows you to associate all of the files for a given test.

Vmstat is a tool supplied by nearly every UNIX operating system, so it's generally available, easy to use, and very understandable. However, if you have multiple components of your web site installed on the same machine, say an HTTP server and an application server, it's difficult to isolate the CPU consumption of each process using vmstat . This is one reason why we recommend only deploying one major process to each machine while you're testing, so that you can clearly understand the performance characteristics of each component, and quickly determine which is causing a bottleneck. Some UNIX systems also provide more sophisticated analysis tools such as top and monitor , which help you to understand the CPU utilization of different processes on the same box.

Monitoring CPU on Windows Systems

Microsoft provides the System Monitor as part of the Microsoft Management Console. On Microsoft Windows 2000, the function is located under the Administrative Tools Performance menu structure. The System Monitor provides a dizzying array of information about your system. We suggest starting with simple measurements such as your CPU. (If you use multiprocessor systems, make sure you are getting cumulative reports for all your CPUs.)

Figure 12.2 shows a sample output from the System Monitor. This shows CPU utilization fluctuating as high as 100% with an average of approximately 21%. System Monitor, like many monitoring tools, requires some care. Some measurements require significant resources to obtain, and obtaining them may impact the performance of the system you're measuring. Also, monitoring systems remotely with System Monitor tends to require significant resources.

Figure 12.2. Sample Microsoft Windows 2000 System Monitor. Microsoft Corporation 2002. Screen shot reprinted by permission from Microsoft Corporation.

Monitoring CPU with a Test Tool

Some test tools allow you to monitor the CPU of the system under test (SUT) along with other measurements such as throughput, user load, and the like. If your test tool provides this feature, you might find it easier to coordinate CPU utilization with certain phases of your test. Do not forget to take CPU measurements on the database, HTTP server, and other systems. Some tests tools make this easy by allowing you to collect statistics from multiple servers.