An Overview of Performance Tuning | Microsoft Application Center 2000 Resource Kit 2001

Performance tuning is the main activity associated with performance management. Reduced down to its most basic level, tuning consists of finding and eliminating bottlenecks—a condition that occurs, and is revealed, when a piece of hardware or software in a server approaches the limits of its capacity.

Before starting the performance tuning cycle illustrated in Figure 10.1, you have to do some preparatory work that establishes the framework for ongoing performance tuning activities. You should:

Identify constraints—A site's business case determines priorities, which in turn establish boundaries. Constraints, such as maintainability and budget limits, are factors that cannot be altered in search of higher performance. You have to focus performance work on factors that are not constrained.

Specify the load—This involves determining what services the site's clients require and the level of demand for those services. The most common metrics for specifying load are the number of clients, client think time (the delay between when a client receives one reply and when it submits the next request), and load distribution (steady or fluctuating, average, and peak load).

Set performance goals—Performance goals have to be explicit, which involves identifying the metrics that will be used for tuning as well as their corresponding benchmark values. Total system throughput and response time are two common metrics that are used to measure performance. After identifying the performance metrics, you have to establish quantifiable and reasonable benchmark values for each one.

NOTE
Because performance and capacity are so closely related, the constraints, load, and goals that you identify are also applicable to capacity planning

After establishing the boundaries and expectations for performance tuning, you can begin the tuning cycle, which is an iterative series of controlled performance experiments.

The Tuning Cycle

The four phases of the tuning cycle shown in Figure 10.1 are repeated until you achieve the performance goals that you established prior to starting the tuning process. Let's examine each phase, starting with Collecting.

click to view at full size

Figure 10.1 The performance tuning cycle

Collecting

The Collecting phase is the starting point of any tuning exercise. During this phase you're simply gathering data with the collection of performance counters that you've chosen for a specific part of the system. These counters could be for the network, the server, or the back-end database.

Regardless of what part of the system you're tuning, you require a baseline against which to measure performance changes. You need establish a pattern of system behavior when the system is idling as well as when specific tasks are executed (for example, adding a member to the cluster and synchronizing it to the controller). Therefore, your first data-gathering pass is used to establish a baseline set of values for the system's behavior. The baseline establishes the typical counter values that you'd expect to see when the system is behaving satisfactorily.

NOTE
Baseline performance is a subjective standard—you have to set a baseline that's appropriate for your work environment and that best reflects your system's workload and service demands.

Once you've established your baseline, you can apply load to the system by using a tool such as Web Application Stress (WAS) to simulate user load.

NOTE
Test scripts should try to mimic typical client usage patterns on your system and will use the load factors that you established earlier: number of concurrent connections, think time, and load distribution.

Analyzing

After you've collected the performance data that you require for tuning the part of the system that you're working on, you need to analyze the data to determine where the bottleneck is. Remember, a performance number is only an indicator—it doesn't necessarily identify the actual bottleneck because a performance problem can be traced back to multiple sources. It's also not uncommon for problems in one system component to be the result of problems in another component (a memory shortage is the best example of this; it's indicated by increased disk and processor use).

The following points, taken from the Microsoft Windows 2000 Resource Kit, provide guidelines for interpreting counter values and eliminating false or misleading data that might cause you to set inappropriate target values for tuning.

Monitoring processes of the same name—Watch for unusually large values for one instance and not the other. Sometimes, the System Monitor misrepresents data for separate instances of processes of the same name by reporting the combined values of the instances as the value of a single instance. You can work around this by tracking processes by the process identifier.
Monitoring several threads—When you are monitoring several threads and one of them stops, the data for one thread might appear to be reported for another. This is because of the way threads are numbered. You can get around this by including the thread identifiers of the process's threads in your log or display. Use the Thread\Thread ID counter for this purpose.
Intermittent spikes in data values—Don't give too much weight to occasional spikes in data. These might be due to the startup of a process and are not an accurate reflection of counter values for that process over time. Counters that average, in particular, can cause the effect of spikes to linger over time.
Monitoring over an extended period of time—We recommend using graphs instead of reports or histograms because the latter views only show the last values and averages. As a result, you might not get an accurate picture of values when you're looking for spikes.
Excluding start-up events—Unless you have a specific reason for including start-up events in your data, exclude these events because the temporarily high values they produce tend to skew overall performance results.
Zero values or missing data—Investigate all occurrences of zero values or missing data. These can hamper your ability to establish a meaningful baseline.

Configuring

After you've collected your data and completed the analysis of the results, you can determine which part of the system is the best candidate for a configuration change and implement this change.

The cardinal rule for implementing changes is only implement one configuration change at a time. A problem that appears to be related to a single component might be the result of bottlenecks involving several components. For this reason it's important to address problems individually. If you make multiple changes simultaneously, it may be impossible to accurately assess the impact of each change.

Testing

After implementing a configuration change, you'll have to complete the appropriate level of testing to determine the impact of the change on the system that you're tuning. At this point, it's a matter of determining whether or not the change:

Improved performance. Did the change improve performance, and if so, by how much?
Degraded performance. Did the change cause a bottleneck somewhere else?
Had no impact on performance. Did the change have any noticeable impact at all on performance?

If you're lucky and performance improves to the level you anticipated, you can quit. If not, you have to step through the tuning cycle again.

Testing do's

Check the correctness and performance of the application that you're using for testing by looking for memory leaks and inordinate delays in response to client requests.

Ensure that all the tests are working correctly.

Make sure that all the tests can be repeated by using the same transaction mix and the same clients generating the same load. (See "The Web Application Stress Tool," later in this chapter.)

Document changes and results.

TIP
You can obtain the monitoring results of your testing from monitoring log files—which can be exported to Microsoft Excel—and the Event log.

Before—and during—performance tuning, we recommend that you use the following two resources for performance monitoring and tuning: the Microsoft Windows 2000 Server Resource Kit and Appendix C, "The Art and Science of Web Server Tuning with Internet Information Services 5.0." These two resources provide most of the information that you will need to successfully tune your network, servers, and Web servers.

Microsoft Windows 2000 Server Resource Kit: Server Operations Guide

The performance monitoring section of the Server Operations Guide contains the following topics and subtopics:

"Overview of Performance Monitoring":
"Performance Monitoring Concepts"
"Monitoring Tools"
"Starting Your Monitoring Routine"
"Analyzing Monitoring Results"
"Investigating Bottlenecks"
"Troubleshooting Problems with Performance Tools"
"Specific Monitoring Scenarios"
"Monitoring Legacy Applications"
"Integrating the System Monitor Control into Office and Other Applications"
"Evaluating Memory and Cache Usage":
"Overview of Memory Monitoring"
"Determining the Amount of Installed Memory"
"Understanding Memory and the File System Cache"
"Optimizing Your Memory Configuration"
"Establishing a Baseline for Memory"
"Investigating Memory Problems"
"Analyzing Processor Activity":
"Overview of Processor Monitoring and Analysis"
"Establishing a Baseline for Processor Performance"
"Recognizing a Processor Bottleneck"
"Processes in a Bottleneck"
"Threads in a Bottleneck"
"Advanced Topic: Changing Thread Priority to Improve Performance"
"Eliminating a Processor Bottleneck"
"Examining and Tuning Disk Performance":
"Disk Monitoring Concepts"
"Configuring the Disk and File System for Performance"
"Working with Disk Counters"
"Establishing a Baseline for Disk Usage"
"Investigating Disk Performance Problems"
"Resolving Disk Bottlenecks"
"Evaluating Cache and Disk Usage by Applications"
"Monitoring Network Performance":
"Introduction to Network Performance Analysis"
"Tools for Monitoring Network Performance"
"Resolving Network Bottlenecks"
"Measuring Multiprocessor Activity":
"Overview of SMP Performance and Monitoring"
"Monitoring Activity on Multiprocessor Systems"
"Optimizing and Tuning Multiprocessor Installations"
"Application Design and Multiprocessor Performance"
"Network Load Balancing and Scaling"

Appendix C, "The Art and Science of Web Server Tuning with Internet Information Services 5.0"

This white paper, written by George Reilly at Microsoft, is arguably the best single resource for tuning a server that's running IIS. This document covers the following topics and subtopics:

"Why Tune Your Web Servers?"
"What to Tune"
"Monitoring Your Hardware"
"Security"
"Monitoring Your Web Applications"
"Tuning Your Web Applications"
"Tools to Monitor and Test Server Performance"
"Features and Settings in Windows 2000 and IIS 5.0"
"Tuning and Troubleshooting Suggestions"
"Testing, Piloting, and Going Live"

This white paper also provides a wealth of information, such as optimal IIS metabase and registry settings, Windows 2000 optimization tips, and ASP caching. Because this document is updated on a regular basis, you should check fo the most recent version at http://www.microsoft.com/technet/iis/default.asp.