Other Performance Monitors

Other Performance Monitors

Earlier we discussed CPU utilization as one of the key measurements of your web site. Now let's consider some other key measurements to help you better understand the performance of your web application.

Network Monitoring

We discussed in Chapter 9 how to estimate network capacity for your web site. During the test, however, you need to validate these estimates, particularly if your web site demonstrates bottleneck symptoms such as underutilization. Regrettably, networks don't come equipped with a handy external gauge to report how much traffic they're handling. Instead, you must rely on a variety of tools to tell you how much traffic goes through your network.

On some systems, you may use the netstat command to monitor network traffic from a machine attached to an Ethernet network. The machine "sees" all of the traffic passing across the network from its network interface card and tells you how many packets passed through the NIC. Take the netstat measurement before starting the test and after it completes. The difference is the amount of traffic moved across your network during the test. Confirm that the number of packets sent and received looks somewhat reasonable. Also notice whether you're losing a relatively large number of packets over the course of the run.

For Windows systems, use the following command to obtain network statistics:

 netstat e 

For Solaris and AIX systems, use this command:

 netstat -in 

Some sample output from the netstat “e command is shown in Listing 12.4. If you see a proportionally large number of errors or discards in the Received or Sent columns , check your network as well as the sending/receiving machine. Discards indicate dropped packets and point to either network congestion, problems with a specific sending/receiving machine, or a poorly configured network. We cannot emphasize this enough: You cannot start testing until you successfully analyze and fix any network issues.

Listing 12.4 Sample netstat output

 C:\>netstat -e Interface Statistics                           Received          Sent Bytes                    354827077       7435073 Unicast packets             239280        126297 Non-unicast packets          31895           127 Discards                         0             0 Errors                           0             0 Unknown protocols            12062 

If you need more detailed network utilization, consider using a network protocol analyzer to analyze your network. The functionality of these tools increases proportionally with price, of course. Some network protocol analyzers exist as freeware software programs and provide some basic information on network capacity, but usually not detailed traffic breakdowns. Also, these inexpensive or free software tools tend to support the slower networks rather than the newer , faster networks.

If you really need detailed information about your network, or you're using a high-speed network, you might require a dedicated network analyzer. These specialized pieces of equipment usually cost thousands of dollars, but they make it much easier to diagnose tricky network problems. Appendix C provides a list of some network protocol analyzer vendors .

These tools examine the packets as they travel across the network and formulate average response times for each component in the test environment. For instance, if the packets for a database request all enter the database server, but the first response packet returns several seconds later, consider tuning the database server. Almost all protocol analyzers provide some analysis of network congestion. Many also help in tracking down misconfigurations, such as DNS problems.

Warning: Check with your network security office before using a network protocol analyzer on your company's network. Many companies actively monitor for network protocol analyzers and consider them a security risk. Also, some companies immediately terminate any employees using these devices on an internal network. Please take this warning seriously, and ask before you use one of these devices.

Software Logs

Several times a year, we receive a call from a customer with a potential performance problem. The customer has tuned a variety of things in the system but can't resolve the bottleneck. Frequently, after examining the log files from the systems involved, we find application errors contributing to the bottleneck condition.

Monitor the logs from your system during each run. As we discussed earlier, the performance test might be the first true, multi-threaded test for a given application. Errors under load occur frequently and usually show up in the logs. Also, the logs sometimes reflect problems due to outages or errors from remote systems. Review the logs after each run, even if things seem to be going well. (Remember, some errors artificially improve test performance.)

Save the logs using a naming convention to distinguish the logs from each run (as discussed earlier in the section on CPU utilization). Keep at least a few days' worth of logs around in case you need to review them later. Of course, watch your hard disk space, particularly on the systems under test. If necessary, archive the logs to a system not actually involved in the testing.

Finally, set your logging to production levels unless you're chasing a problem. We often find customers using verbose logging during their performance testing, which skews their performance numbers . After your initial runs, turn the logging down to the levels you plan to use in production. Otherwise, you impact your performance with excessive disk I/O.

Java Application Server Monitors

Some Java application servers allow you to monitor key measurements as the application server executes. For example, the IBM WebSphere Application Server Resource Analyzer tool allows you to monitor the resources inside the running JVM.

Using the Resource Analyzer, you specify the items of interest, such as a connection pool, or the size of a Bean pool inside an EJB container. The Resource Analyzer allows you to watch these items using a graphical interface. Figure 12.5 shows a Resource Analyzer screen capture for the database connection pool. In this time slice, you see that the pool's utilization hovers in the 40% “70% range. In addition, the data shows connections available immediately for threads (concurrent waiters and average wait time both at zero). The database connection pool appears to be behaving well and does not indicate a bottleneck. (If anything it might even be a little too big.)

Figure 12.5. Sample Resource Analyzer output. IBM Corp. 2002. Reprinted by permission of IBM Corp.

If your Java application server provides a runtime monitoring tool, consider using it as part of your testing. These monitors allow you to observe the internals of your web application server during the test. However, keep in mind that some items might be "performance expensive" to monitor. IBM's Resource Analyzer, for example, gives you the relative "weight" of each item you choose to monitor. During performance runs, try to avoid the expensive measurements unless you're trying to obtain more details for problem resolution. You might consider turning off the monitoring tools periodically to make sure they're not impacting your performance.