Profiling with Performance Counters

Performance counters the basic unit of measurement for gauging the performance of your application were introduced in Chapter 12, when we considered connection pooling. However, a lot more information is available from the performance monitor than the number of open connections, and this information can prove tremendously useful when you're trying to improve an application's performance. Performance counters are particularly useful when you're attempting to identify a bottleneck that's hampering application performance.

To open the Performance window, choose Performance from the Administrative Tools section in Control Panel. By default, you'll see only performance counters for measuring basic information such as the computer's CPU and disk drive use. However, .NET installs much more useful counters for tracking application performance. To add a new counter, right-click on the Counter list, choose Properties, and click on the Data tab. Before continuing, remove all the default counters. You can then click Add to insert some more useful ones (as shown in Figure 14-8).

Note that you can specify a computer name in the Add Counters window. This enables you to monitor the performance of a remote computer without incurring additional overhead that could skew the results.

Figure 14-8. Adding performance counters

Essential Performance Counters

On an average computer, dozens of performance counters are available. Examples include data source specific counters (such as those included with SQL Server), the ten basic .NET performance categories, two ASP.NET categories, and hardware-specific categories that profile memory or CPU usage and disk activity. Clearly, it's not easy to sort out which counters are best for profiling system activity. It can help to refer to the descriptions of some of the most important categories and counters in Table 14-6. You also can refer to the MSDN Help for a full description of every .NET counter.

Table 14-6. Some Valuable Performance Counter Categories 

Category	Description
Processor	Studying the % CPU Utilization counter can give you a basic idea of where a bottleneck lies. If your application throughput (requests to server per second) levels off while the CPU utilization is still small (less than 70 percent), a bottleneck exists that isn't related to CPU speed.
.NET CLR Data	Tracks open and pooled connections with the SQL Server data provider. Useful for determining pool usage, in conjunction with any available data source provider counters.
.NET CLR Exceptions	The # of Exceps Thrown counter is a basic measure of health. If it rises unexpectedly, it might indicate that error conditions are being met, although it might also indicate the activation of normal exception handlers in response to a common problem or an invalid user action.
.NET CLR Locks and Threads	Tracks thread contention (more than one thread competing for a locked resource, forcing at least one thread to wait). If you're implementing your own threading, as described in Chapter 7, you need to become familiar with these counters.
.NET CLR Memory	Provides information related to .NET garbage collection. This might indicate a problem if large objects are being created and released frequently and are wasting memory before they are garbage collected. Unfortunately, this sort of problem usually can't (and should not) be dealt with manually.
.NET CLR Remoting	Provides useful .NET Remoting counters, such as Remote Calls/sec and Total Remote Calls. These counters can be global or relative to a single component host application (which you must indicate).
.NET CLR Security	Provides information about code access security checks and enables you to determine the performance penalty associated with a check. These counters aren't often used for improving performance, but they can help you understand code access security.
ASP.NET	Useful with XML Web service applications. The Requests Queued counter indicates the number of requests waiting to be processed by an XML Web service. You can study this counter, while changing the workload, to determine the maximum throughput your XML Web service can support. Application Restarts and Worker Process Restarts can indicate whether general health is poor. Remember, however, that the application will be restarted if configuration settings change, and the worker process will be recycled periodically according to the options set in the <processModel> section of the machine.config file.
ASP.NET Applications	Similar to the ASP.NET category, but you can choose a single XML Web service application to profile. Requests/sec indicates the application throughput, and Pipeline Instance Count indicates the maximum number of concurrent requests being served. Several counters also enable you to test data caching (including Cache API Hits, Cache API Hit Ratio, and Cache API Turnover Rate) and output caching (Output Cache Hits and Output Cache Turnover Ratio).

You might have noticed one puzzling fact: There are four ASP.NET categories, and some of them seem similar. You'll find an ASP.NET category, which provides information about the overall performance of ASP.NET, and an ASP.NET Applications category, which provides information about a single specified Web application. You'll also find two extremely similar categories with version information (ASP.NET [version] and ASP.NET Apps [version]). These categories are actually the same as the versionless categories.

Technically, the versionless categories map to the latest version of the .NET Framework. If you install multiple versions of the .NET Framework, you will find one ASP.NET [version] category and one ASP.NET Apps [version] category for each version of the framework. However, the ASP.NET and ASP.NET Application categories automatically represent the latest installed version.

Now That I'm Monitoring, What Does It Mean?

As Microsoft tactfully states, "Deciding whether or not performance values are acceptable is a subjective judgment that varies significantly with variations in user environments. The values you establish as the baseline for your organization are the best basis for comparison."

In other words, many numbers don't mean anything, at least not without context. A 90 percent CPU level means your CPU is being used heavily. However, the significance of this fact depends on the test conditions, including the hardware and the current user load. In fact, many of these numbers are most important when they deviate from the expected values. Before evaluating them, you must drive your system through numerous test scenarios and develop a sense of the normal operating ranges for your equipment. You should carefully record these typical values. When you make changes to the system or hardware in the future, you can use these values as a baseline.

Custom Performance Counters

.NET provides classes in the System.Diagnostics method that enable you to interact programmatically with Windows performance counters, retrieving the current set of counters, retrieving the current value for a specific counter, or creating and incrementing your own counters. This section examines these tasks.

The first step when you use custom counters is to create the counters and counter category. You can do this manually in Visual Studio .NET using Server Explorer or programmatically using the classes in the System.Diagnostics namespace. You can even create a custom installer as part of a Visual Studio .NET setup project, which adds the counters when the application is installed on a computer.

Listing 14-13 shows the code needed to create a custom counter category (provided it doesn't already exist) and add two counters.

Listing 14-13 Creating custom counters

 If Not (PerformanceCounterCategory.Exists("Shopping Counters")) Then     Dim Counters As New CounterCreationDataCollection()     ' Create a simple counter that records a total value.     Dim CountCounter As New CounterCreationData()     CountCounter.CounterName = "Total Purchases"     CountCounter.CounterHelp = "Total number of purchases " & _                                "submitted by this component"     CountCounter.CounterType = PerformanceCounterType.NumberOfItems32     ' Create a rate counter that records a per second value.     Dim RateCounter As New CounterCreationData()     RateCounter.CounterName = "Purchases/Second"     RateCounter.CounterHelp = "Current number of purchases per " & _                               "second submitted by this component"     RateCounter.CounterType = RateOfCountsPerSecond32     ' Add both counters to the collection.     Counters.Add(CountCounter)     Counters.Add(RateCounter)     ' Create the custom category.     PerformanceCounterCategory.Create("Shopping Counters", _       "Counters for the Shopping.dll component", Counters) End If 

The CounterType property identifies the type of counter that you're creating. Although .NET provides several options, most choices are concurrency counters and rate counters. Concurrency counters reflect a simple value. When your program increments a concurrency counter, the new value appears identically in the performance monitor. Rate counters involve an extra calculation. You set rate counters in the same way that you set concurrency counters. Behind the scenes, however, these values are cached for a second. The current value of the counter is always equal to the total count received in the last second.

The 32 at the end of the counter type indicates the size allocated to store the counter data. If you expect to require values larger than a 32-bit integer, you can use the corresponding type that ends with 64. You also can use more advanced counters such as AverageCount64 or CounterDelta32, which calculate values by comparing the number of counts in a given interval with the number of counts in a previous interval. The formulas used for these calculations are outlined in the MSDN Reference.

When you create counters, it's also a good idea to include some descriptive information. In Listing 14-13, this includes the CounterHelp string assigned to each counter and a similar category description. The user can view this information when adding a counter in the Performance window (as shown in Figure 14-9).

Figure 14-9. Adding custom performance counters

As you might imagine, .NET makes it similarly easy to remove a counter category:

 PerformanceCounterCategory.Delete("Shopping Counters") 

If you prefer to create counters at design time with Visual Studio .NET, you can just follow these steps:

1. Open Server Explorer, and expand the node for the server you want to view. Right-click on the Performance Counters node and choose Create New Category.

2. Enter a name and description for the category you want to create in the Performance Counter Builder dialog box.

3. In the New Counter frame, specify the counter name, type, and description. Click Add To List.

4. Repeat step 3 for each new counter you want to create in this category.

To use a counter, you need to create a counter instance. Counter instances relate to performance counters in the same way that objects relate to classes. Although there is only one Purchases/Second counter defined, a server component can create multiple instances. Depending on your needs, a counter might be created for every instance of a server-side component, or a single server-side component might create multiple instances to represent work it is doing for different clients.

Creating a performance counter is easy. You just create an instance of the PerformanceCounter class and specify the counter category, counter name, and the instance name. You must also add the optional False parameter; otherwise, the counter will be created in read-only mode. (You also can access a counter on another computer by specifying a machine name parameter, but only in read-only mode.)

For example, here's the code that creates an instance of the total counter and increments it by one:

 Dim TotalCounter As New PerformanceCounter("Shopping Counters", _   "Total Purchases", "NewCustomerInstance", False) ' Increment the counter by 1. TotalCounter.Increment() 

You also can set the PerformanceCounter.RawValue property directly, but the Increment method ensures that the operation is atomic. This way, multiple threads or components can increment the same counter at once without losing data. Figure 14-10 shows the result in the Performance window after the total counter has been incremented several times.

Figure 14-10. Incrementing a custom concurrency counter

You can adjust the rate counter just as easily. If you monitor the performance counter while executing the following code, however, a very different behavior results: The counter spikes to 100 once the code is executed, but falls back rapidly, reaching 0 one second later, as the rate of counts/second dwindles.

 Dim RateCounter As New PerformanceCounter("Shopping Counters", _   "Purchases/Second", "NewCustomerInstance", False) ' Increment the counter by 100. RateCounter.IncrementBy(100) 

Figure 14-11 shows the results of executing this code multiple times.

Figure 14-11. Incrementing a custom rate counter

When the application ends or when the counters are no longer needed, be sure to release them. Otherwise, the counters remain until the Performance window is closed.

 RateCounter.RemoveInstance() TotalCounter.RemoveInstance() 

Stress Testing

Performance counters help you gauge how your applications perform; to draw meaningful conclusions, however, you need to study how these counters respond under different loads. Unfortunately, .NET doesn't provide any easy tool to automate this task for you. Although stress-testing tools are available (such as Application Center Test, which is included with some versions of Visual Studio .NET, and its predecessor, Web Application Stress Tool (WAST), which is available for a free download from http://webtool.rte.microsoft.com), few tools support .NET Remoting or XML Web services directly. To simulate a realistic load, you will probably need to create your own testing application.

The basic strategy is to design a multithreaded application that makes continuous requests on multiple different threads. You should start with a modest load and gradually increase the number of threads until the application begins to generate timeout errors or manifest other problems. Then step back and use the highest load that doesn't cause any problems.

After you've found the highest load your application can support, the next (and more difficult) step is to determine what the current bottleneck is. One rule of thumb is that if the CPU usage is at 85 percent or more, the CPU is likely to be the bottleneck. If the maximum number of simultaneous requests is reached before this point, the bottleneck lies elsewhere, such as with memory, configuration, thread contention, hard drive speed, or database locking. Discovering this bottleneck requires practice, patience, and performance counter monitoring.

When performing stress testing, you should use a dedicated Web server and a set of dedicated client machines. These client machines should interact with the server over a fast isolated network. If you try to test the server using a load-testing tool running from the same computer, you will retrieve data that is much less accurate and much less useful.