High CPU Utilization

During your testing, the CPU utilization of one of your servers may reach high levels (95% or above). High CPU utilization indicates CPU saturation. You have exhausted the processing capacity of this computer.

What are your options after the application server CPU reaches full utilization? Is this the maximum load for your server, or can you add more load to this server? To answer these questions, we explore the components of the burdened CPU. We also look at the throughput plateau and corresponding response times of the busy system.

High User CPU

In Figure 13.4, the total CPU contains a high amount of user CPU (68% in this case). High user CPU means the server is very busy handling requests from the web application. Also, notice that the system component of CPU in our example is at 30%. For Java applications, the system component of CPU normally tends to be somewhat high.

Figure 13.4. Example of high CPU utilization. From "Experiences with Successful Website Performance Testing," presented by Stacy Joines at the Solutions 2001 conference, San Francisco, CA. IBM Corp. 2001. Reprinted by permission of IBM Corp.

If the application uses most of the CPU, consider these choices before continuing with your testing. First, consider your response time. For a reasonably tuned application, the CPU saturation indicates a throughput plateau for your testing on this server. In other words, you cannot continue to increase throughput on this server regardless of how much additional load you apply.

However, the CPU saturation does not necessarily mean the server cannot support additional load. Users only know response time. A saturated server may continue to support increasing load as long as the response time remains acceptable. Of course, the server queues the arriving load once the server becomes saturated . However, if the wait in the queue remains short, the overall response time remains acceptable as well. (See Chapter 1 for more discussion on throughput, load, and response time dynamics.)

Under normal conditions, we don't recommend increasing the load on a saturated CPU. The additional load waits for CPU resources to free. If load increases on this server, the wait time grows linearly for all users. As a general rule, try to keep queues in front of your HTTP server to allow optimal load balancing. Minimizing queues within your application and database server provides the optimal performance. However, your test results may show that the high user CPU does not prevent you from achieving your performance objectives.

Let's assume you anticipated getting better throughput or response time through the server than you're currently achieving. Let's look at some of the common causes of high CPU utilization during a test run.

How to Diagnose

During your tests, the CPU utilization reaches 100%. If the user component of CPU exceeds 75%, with little or no utilization in the wait component, the system runs with high user CPU utilization.

How to Resolve

• Tune the application. Make the application more efficient so it requires less CPU to perform the tasks you're requesting of it. Use a code profiler to uncover inefficiencies in the application and make improvements.

  To help focus on the critical paths to profile, look at a "runtime profile" of your application. When your system is running under load, which areas of the web application does the profiler identify as being slow? Focus your tuning efforts on the slow but frequently called areas of your application.

  After completing your profiling exercise, consider using thread traces to further remove bottlenecks in the application. The thread trace shows the thread activity under load, unlike many code profilers, which show only a single execution path .

  Application tuning makes a lot of sense when you reach CPU saturation. You might find opportunities for significant performance improvements and reduce the CPU burden of your application. This increases the capacity of each of your servers and reduces the equipment you need for your web site.

• Add processing capacity. At some point, additional application tuning begins to provide diminishing benefits. If you need to support additional load, you need additional processing capacity. Try adding servers to the web site cluster or increasing the CPUs in the server. You might also try increasing the speed of your CPUs by updating the web site hardware. Refer to Chapters 1 and 3 for more discussion on scaling a web site.

High System CPU

As mentioned in the previous section, the system component of CPU tends to run somewhat high with Java applications. However, other factors influence system CPU. Too much time devoted to this component impacts the CPU capacity available to run the web application. Figure 13.5 shows the application server at 80% CPU utilization. Of this, the system component absorbs 45% of the utilization.

Figure 13.5. Example of high system CPU utilization. From "Experiences with Successful Website Performance Testing," presented by Stacy Joines at the Solutions 2001 conference, San Francisco, CA. IBM Corp. 2001. Reprinted by permission of IBM Corp.

How to Diagnose

Observe the CPUs on your application server. If your CPU is near 100%, look at the breakdown between user, system, and wait time. Check to see if the system component is using more than 30% of the total CPU, and there is little to no wait time.

How to Resolve

• Check the HTTP server. If the HTTP server runs on the same machine as the application server, the listeners started by the HTTP server impact the CPU of the machine. This impact usually appears in the system component of the CPU utilization. To resolve this problem, reduce the number of listeners started by the HTTP server or move the HTTP server onto its own machine.

  Excessive listeners (over a few hundred, for example) usually indicate a performance problem within the web application. If the application responds quickly, the web site usually requires fewer listeners.

• Check other processes running on the box. These processes sometimes run at a low level in the system and impact system CPU. Eliminate unnecessary processes on your application server boxes.

High Wait CPU

Excessive wait CPU indicates a serious performance problem, and it usually occurs when the CPU interacts with a slower resource. Usually the high wait cycle problem emerges when the server spends a proportionally large amount time interacting with the hard disk. However, occasionally high I/O wait times result from network problems or other issues.

Any machine is prone to high wait CPU, although it most commonly occurs on database and web application servers. Figure 13.6 shows the database server at 75% CPU utilization, with 50% of this devoted to wait time.

Figure 13.6. Example of high CPU wait on a database server

Note that all database servers usually show some amount of CPU wait. However, if this component becomes too high, it shows that the CPU accomplishes very little aside from waiting on the hard disk. In these cases, tuning is usually appropriate.

How to Diagnose

• Observe the CPUs on all your servers. Look for any database or other server performing extensive I/O with more than 30% wait time, and any less I/O intensive server with greater than 3% wait time.

• Use a thread trace at the application server to determine if the I/O really represents a bottleneck. If you see threads waiting for logging to complete or waiting for a database with high wait CPU, you need to begin tuning.

How to Resolve

• Check application logging. Excessive application logging requires the system to interact with the hard drive more frequently, which triggers abnormally high CPU wait.

  We've discussed effective logging strategies for various components throughout the book and won't repeat them here. However, review these sections to develop a better logging strategy. (In case you've forgotten, Chapter 9 discusses database logging, Chapter 2 covers web application logging, and Chapter 3 discusses the HTTP server logs.)

  Test clients also succumb to excessive logging problems, so keep your logging at the test client to a minimum (particularly if your test requires hundreds or thousands of virtual users).

• Check disk space availability on the server. If the server doesn't have enough room to write the logs, you might see CPU wait problems. You might also see application errors. Add disk space, reduce logging, or clean out some space on the existing drives .

• Check database efficiency. If the system in question is a database server, try some of the tuning tips for databases mentioned in Chapter 9. Consult your DBA for detailed reports on database performance and consider tuning the database to make its disk I/O more efficient.

• Check remote systems and the network. In rare cases, sluggish remote systems trigger high wait CPU. After you eliminate other causes, check the performance of the remote systems and the network connected to your web site.