11.6 Performance Monitor | Monitoring and Managing Microsoft Exchange Server 2003 (HP Technologies)

< Day Day Up >

You can do some basic monitoring of processes and queues using each Ex-change server's Monitoring tab. However, many other key resources should be regularly monitored. Windows includes a Performance Monitor that can be used to monitor these other Windows and Exchange resources, such as the number of Internet messages received per minute, the percentage of processor use, and so on. Performance Monitor can create charts, set alerts, and format reports that help the Exchange administrator measure system performance. Data gathered from counters can be displayed in real-time or stored in log files for later analysis.

The usefulness of any monitoring effort depends on creating a baseline measurement when the system is operating effectively and knowing the limits-both high- and low-end values-for each parameter that indicate a problem. The values in Windows Performance Monitor counters can be examined to determine how Exchange is performing or to track error conditions.

Performance Monitor tracks the value of object counters, where objects include the system's processors, memory, disks, and processes, such as the Exchange processes. Each of these object types has a set of counters from which the Performance Monitor can collect data. For example, a Logical Disk object has counters for % Disk Time, Free Megabytes, and % Free Space.

Some object types have several instances. For example, the system's Processor object type will have multiple instances if a system has multiple processors. The PhysicalDisk object will have one instance for each disk drive. The counters for each instance of an object can be monitored independently.

Most typical monitoring can be done using the Windows Performance Monitor. Using Performance Monitor, the Exchange administrator can track the functioning of critical objects on an Exchange server. By carefully watching the monitored objects, the administrator often can detect a minor problem before it progresses into one that will affect users because of server downtime.

The administrator can create Microsoft Management Console files (.MSC) that define specific objects to be monitored. Once created, the .MSC file contains all the settings, including screen position, chart colors, and other settings required to monitor the object. If alerts have been configured, the alert will be sent and the event details will be written to the Windows Event Viewer log when the thresholds are exceeded.

11.6.1 Creating a real-time performance monitor chart

The Performance Monitor's chart view provides a continual, real-time overview of the system performance. The following procedures can be used to create a performance monitor chart.

Start the Performance Monitor from the Windows Start menu by selecting All Programs →Administrative Tools →Performance (Figure 11.13).

Figure 11.13: Performance Monitor window
Select the View Chart button.
Select the Add button to display the Add Counters dialog box (Figure 11.14).

Figure 11.14: Performance Monitor - Add Counters dialog box
Select the Exchange server to be monitored. You can monitor the local system by selecting Use local computer counters. To monitor another server in your network, select Select counters from computer and select the system from the associated drop-down list.
Use the Performance object drop-down list to select an object to monitor. Recommended objects to monitor are listed later in this chapter.
Select the counters and instances to be monitored. The set of available counters varies depending on the object. Use the Instance list to select which instance of an object is to be monitored. For example, the Processor object type will have multiple instances if a system has multiple processors. You can select All counters to monitor all counters for an object, select All instances to monitor all instances, or both. You can select the Explain button to display an explanation of the counter. When you have selected the counters and instances, select Add.
Repeat the previous steps to add additional counters to the chart. Select Close when all desired counters have been added to the chart.
To save these settings for subsequent use, select Save As from the File menu. Provide a file name in the Save As dialog box. The .MSC settings file will save all Performance Monitor settings.

11.6.2 Recording performance data for later analysis

The Performance Monitor can be used to collect and record data over a period. These logs can later be analyzed to identify long-term trends or to troubleshoot problems. The following procedure can be used to create a Performance Monitor log file.

Start the Performance Monitor from the Windows Start menu by selecting All Programs →Administrative Tools →Performance.
Expand the Performance Logs and Alerts item in the MMC tree pane (i.e., the left pane in the window), then select the Counter Logs item (Figure 11.15).

Figure 11.15: Performance Monitor - Counter Logs window
Right-click in the details pane (i.e., the right pane), and select New Log Settings.
Enter a name into the New Log Settings dialog box and select OK to display the log settings dialog box. The title of this dialog box will match the file name you entered into the New Log Settings dialog box (Figure 11.16).

Figure 11.16: Log Settings dialog box
General tab Select Add Counters to display the Add Counters dialog box (Figure 11.17).

Figure 11.17: Add Counters dialog box
Select the Exchange server to be monitored. You can monitor the local system by selecting Use local computer counters. To monitor another server in your network, choose Select counters from computer and select the system from the associated drop-down list.
Use the Performance object drop-down list to select an object to monitor. Recommended objects to monitor are listed later in this chapter.
Select the counters and instances to be monitored. The set of available counters varies depending on the object. Use the Instance list to select which instance of an object is to be monitored. For example, the Processor object type will have multiple instances if a system has multiple processors. You can select All counters to monitor all counters for an object, select All instances to monitor all instances, or both. You can select the Explain button to display an explanation of the counter. When you have selected the counters and instances, select Add.
Repeat the previous steps to add additional counters. Select Close when all desired counters have been added.
On the General tab (see Figure 11.16), set the Interval value to the desired interval for collecting sample data. When collecting data to analyze performance issues, this should be as low as reasonably possible.
Use the Run As field and associated Set Password button to collect data under different accounts. For example, if you need to collect data from a remote server, you can use the Run As field to specify an account that has the necessary permissions on the remote server.
Log Files tab Select the Log Files tab (Figure 11.18).

Figure 11.18: Log Files tab
Use the Log file type drop-down list to select the log file type. The available options are Text File (Comma delimited), Text File (Tab delimited), Binary File, Binary Circular File, or SQL Database. Unless you have another preferred choice, Text File (Tab delimited) is a good choice because it allows you to easily import the collected data into Excel for further analysis.
Use the Configure button to identify the directory where the log file will be stored, the name of the log file, and the log file size limit. Make sure you select a disk drive with sufficient space to hold the log file. If you collect data every 15 seconds, your system can easily use several gigabytes of storage space over a weekend.
You can use the End file names with field and Start numbering at field to append sequence numbers to the end of the log file names.
Schedule tab Select the Schedule tab (Figure 11.19).

Figure 11.19: Schedule tab
Use the Start log fields to enter the time when logging operations should begin.
Use the Stop log fields to enter the time when logging operations should stop. Logging can be stopped automatically after a specified duration (e.g., 1 day), stopped at a specific time, or stopped manually using the shortcut menu.
You can use the When a log file closes options to specify the action that Performance Monitor should take when a log file closes. If you want to configure continuous (automated) logging, select Start a new log file. If you want to run a program when the log file closes (e.g., to copy the just-closed log file to an archive), select Run this command and enter the name of the program to run.
When you have entered all information, select OK to create the logging entry.
To save these settings for subsequent use, select Save As from the File menu. Provide a file name in the Save As dialog box. The .MSC settings file will save all Performance Monitor settings.

11.6.3 Setting alerts using Performance Monitor

The Performance Monitor's alert view provides for setting thresholds on counters. When a counter's threshold is exceeded, the date and time of the event are recorded in the Alert window.

The following procedures can be used to specify alerts using Performance Monitor.

Start the Performance Monitor from the Windows Start menu by selecting All Programs →Administrative Tools →Performance.
Expand the Performance Logs and Alerts item in the MMC tree pane (i.e., the left pane in the window), and select the Alerts item (Figure 11.20).

Figure 11.20: Performance Monitor - Alerts window
Right-click in the details pane (i.e., the right pane), and select New Alert Settings.
General tab Enter a name into the New Alert Settings dialog box and select OK to display the alert settings dialog box (Figure 11.21). The title of this dialog box will match the file name you entered into the New Alert Settings dialog box.

Figure 11.21: Alert Settings Properties - General tab
On the General tab, select Add to display the Add Counters dialog box (Figure 11.22).

Figure 11.22: Add Counters dialog box
Select the Exchange server to be monitored. You can monitor the local system by selecting Use local computer counters. To monitor another server in your network, select Select counters from computer and select the system from the associated drop-down list.
Use the Performance object drop-down list to select an object to monitor. Recommended objects to monitor are listed later in this chapter.
Select the counters and instances to be monitored. The set of available counters varies depending on the object. Use the Instance list to select which instance of an object is to be monitored. For example, the Processor object type will have multiple instances if a system has multiple processors. You can select All counters to monitor all counters for an object, select All instances to monitor all instances, or both. You can select the Explain button to display an explanation of the counter. When you have selected the counters and instances, select Add.
Repeat the previous steps to add additional counters. Select Close when all desired counters have been added. The alert settings dialog box will be redisplayed.
Use the Alert when the value is drop-down list to select the condition to test (either Under or Over), and enter the threshold value for issuing the alert into the Limit field (see Figure 11.21).
Use the Interval field and the Units drop-down list to specify the desired interval for sampling data.
Use the Run As field and associated Set Password button to set an alert using different accounts. For example, if you need to create an alert on a remote server, you can use the Run As field to specify an account that has the necessary permissions on the remote server.
Action tab Select the Action tab to select the actions to be executed when the threshold value for the alert is reached. You may select as many of the actions as needed.
Select the Log an entry in the application event log check box if you want to log an entry when the threshold value is reached (Figure 11.23). You can view entries in the applications event log using the Windows Event Viewer.

Figure 11.23: Alert Settings Properties - Action tab
Select the Send a network message to check box if you want to send a network alert to a specified workstation. Enter the workstation name into the associated field. The alert message will only be delivered if the workstation is turned on, if a user is logged onto the workstation, and the messaging service is running on the workstation. This type of notification should normally be used only in environments where the network is very reliable, such as a dedicated monitoring workstation on the same local area network (LAN) as the monitor.
You can use the Start performance data log check box to start a performance data log you have previously defined.
You can select Run this program to run your own specialized alert program. A special notification application can be used to alert administrators who are not logged on to the network. For example, a notification application can be used to start a pager program to page an Exchange administrator who is not always logged on to the system. Use the Command Line Arguments button to specify the arguments that are to be passed to the special notification application.
Schedule tab Select the Schedule tab (Figure 11.24).

Figure 11.24: Alert Settings Properties - Schedule tab

Use Start scan to enter the time when monitoring operations should begin.

Use Stop scan to enter the time when monitoring operations should stop. Monitoring can be stopped automatically after a specified duration (e.g., 1 day), stopped at a specific time, or stopped manually using the shortcut menu.

You can use the When an alert scan finishes option to specify the action that Performance Monitor should take when a scan finishes. Select the Start a new scan check box to configure continuous alert scanning.

When all information has been specified, select OK to create the alert entry.

To save these settings for subsequent use, select Save As from the Console menu. Provide a file name in the Save As dialog box. The .MSC settings file will save all Performance Monitor settings.

11.6.4 Recommended objects to monitor

Windows and Exchange provide many objects that can be monitored. Monitoring all of the possible objects is unnecessary and, in fact, may adversely affect network and server performance. The objects to monitor most closely are those processes that handle message flow through the system. These are the MTA and the message queues for the connectors. Monitors for other secondary objects can be configured to help diagnose suspected problems.

Several types of counters should be monitored:

The Windows and hardware resources essential to proper functioning of Exchange should be carefully monitored. This includes such counters as the percentage of CPU time being used, the amount of free disk space available on key disk volumes, and other resources.
Another group of counters that must be monitored closely includes those counters that show whether e-mail messages are flowing through the system as expected. These counters are principally the number of messages awaiting processing in various Exchange queues.
A third set of counters includes those items that provide an indicator as to how heavily the e-mail system is being used. This includes counters such as the number of currently active e-mail users, the rate at which messages are being processed, and others. Data from these counters should be collected when the e-mail system first becomes operational, and periodic checks should then be made to track the growth of e-mail usage. These counters can be helpful to justify increasing hardware to support increasing workloads.
There are a great number of Exchange counters that do not fall into any of the three categories listed above. These generally do not need to be closely tracked, but they are often useful for troubleshooting problems.

The administrator should create shortcuts for the .MSC settings file on the system that will be used to monitor the Exchange objects. The complete list of Performance Monitor counters is extensive. The following sections contain only those counters relevant to Exchange and include some recommended .MSC settings files.

Monitoring key resources

The Performance Monitor counters in this section are recommended as the foundation set of counters for all Exchange servers. These counters monitor the Windows and hardware resources essential to proper functioning of Exchange. Additional counters should be added in accordance with the function of the Exchange server. These counters should be combined into a single performance monitor graph with an update interval of about 1 minute.

Note

When you are doing performance monitoring to investigate specific problems, you will usually want to use the lowest possible interval.

Logical Disk

% Disk Time. This counter records the percentage of time a hard drive is either reading or writing. A sustained value greater than 90% indicates that the hard drive is a performance bottleneck. There is one instance for each logical drive.
% Free Space. Trigger an alert if the amount of free disk space available decreases to less than 25%. There is one instance for each logical drive, and each instance should be monitored. However, there is no reason to monitor the _Total instance.

Free Megabytes. This is the amount of free space remaining on a logical drive. It is especially important to monitor the log drive to ensure that it does not fill up with log files. Log files are removed whenever an online normal backup is performed by a backup program that uses the Exchange backup Application Programming Interface. If the log files are not being removed, verify that the backups are being completed successfully.

Note

Before Windows 2000, you had to enable collection of disk monitoring counters using the Windows diskperf -yv command. However, these counters are enabled by default for Windows 2000 and Windows 2003.

Memory

% Committed Bytes in use. This counter is the ratio of Committed Bytes (physical memory in use for which space has been reserved in the paging file) to Commit Limit (determined by the paging file size). Trigger an alert if the use of virtual memory exceeds 80%.
Pages/sec. This counter measures memory paging from/to the virtual memory paging file. A sustained high number of pages per second indicates the need for additional memory. Brief spikes generally do not indicate a problem and can be ignored.

Paging File

% Usage. The paging file usage should generally remain between 15% and 35%. Usage above 60% usually indicates a problem, such as a memory leak or too little RAM. Rebooting the system may provide a short-term, temporary solution in the case of a memory leak. A consistent usage greater than 90% should be considered a critical situation.

PhysicalDisk

Avg. Disk Queue Length. Trigger an alert when the average disk queue exceeds 2.

Process

% Processor Time. The processor time should be monitored for the following instances:
- EMSMTA-Microsoft Exchange MTA Stacks
- INETINFO-Internet Protocols, including Internet Mail Access Protocol 4 and Post Office Protocol 3
- MAD-Microsoft Exchange System Attendant
- STORE-Microsoft Exchange Information Store

The exact list of processes will vary depending on the Exchange components that have been installed on the system.

The % Processor Time is the percentage of elapsed processor time that all of the process threads used. On systems with multiple processors, the maximum counter value is 100% times the number of processors. The processor time for these services should never be at 0% or at the maximum value (i.e., 100% times the number of processors) all of the time. If a process is always at 0%, check All Programs →Administrative Tools →Services to verify that the process is running. If a process is always at the maximum value, check the Event Viewer to identify the problem.

Processor

% Processor Time. This counter records the percentage of time the processor is running nonidle threads and is the primary indicator of processor activity. Servers with multiple processors will have an instance (0, 1, 2, and so on) for each processor. Each instance (i.e., processor) should be monitored. An average value less than 20% indicates the processor is lightly used or services are down. An average value consistently greater than 90% indicates that the processor is being overworked. Trigger an alert if the processor use exceeds 90% for longer than 5 minutes.

Network Interface

Bytes Total/sec. Monitor this counter for each network interface to determine whether network bottlenecks exist. A typical Ethernet interface should not exceed more than 70% use to be considered normal. In a full-duplex environment this number may be increased to 90%. For example, the threshold for a 100-MB Ethernet would be 100,000,000/8 =12.5 MB x 70% = 8.75 MB/second. (This number may be greater than some network interface cards can sustain as all cards are not created equal.)
Packets Outbound Errors. Monitor this counter for any errors. In a properly functioning network this should be zero.
Packets Received Errors. Monitor this counter for any errors. In a properly functioning network this should be zero.

Monitoring message processing

The exact list of objects and counters will vary depending on the Exchange components being used on the system. For example, if the Lotus Notes Connector software is not being used, then the counters for this component will not be available.

The counters listed in this section are those that show whether e-mail messages are flowing through the system as expected. These counters should be combined into a single performance monitor graph with an update interval of about 2 minutes:

MSExchangeIS

RPC Requests. This is the number of client requests that are currently being processed by the Information Store.

MSExchangeIS Mailbox

Receive Queue Size. Trigger an alert if the number of messages in the Private Information Store receive queue is greater than 20.
Send Queue Size. Trigger an alert if the number of messages in the Private Information Store send queue is greater than 20.

MSExchangeIS Public

Receive Queue Size. Trigger an alert if the number of messages in the Public Information Store receive queue is greater than 20. This should be set on public folder servers only.
Send Queue Size. Trigger an alert if the number of messages in the Public Information Store send queue is greater than 20. This should be set on public folder servers only.

MSExchangeMTA

Work Queue Length. This counter is the count of messages in the MTA queues awaiting delivery to other servers or awaiting processing by the MTA. The Work Queue Length should increase and decrease between 0 and 50. When messages are stuck in the queue, the counter will remain level or only increase for extended periods. Watch for 'artificial floors' on the MTA queue.

A high number indicates a probable problem. An alert should be triggered if the Work Queue Length is greater than 100.

Divide this value by the Messages/Sec value to get an estimate of the delay that messages experience when delivered or sent.

MSExchangeNMC

If you are running a Notes Connector, you should monitor the following counters:

Message Queued Inbound. This is the current count of Lotus Notes messages queued at the connector for delivery to Exchange.
Message Queued Outbound. This is the current count of Exchange messages queued at the connector for delivery to Lotus Notes.

MSExchangeSRS

You should monitor the following counters on servers that are running the Microsoft Exchange Site Replication Service if you have a mixed environment that includes Exchange 5.5 servers.

Pending Replication Synchronizations. This counter shows the number of unanswered synchronization requests sent by this server. The synchronization process is complete when the Pending Replication Synchronizations counter and the Remaining Replication Updates counter both reach zero.
Remaining Replication Updates. This counter shows the number of object modifications waiting to be applied to the local server. The synchronization process is complete when the Pending Replication Synchronizations counter and the Remaining Replication Updates counter both reach zero.

SMTP Server

Categorizer Queue Length. This is the current count of messages in the categorizer queue.
Current Messages in Local Delivery. This is the current count of messages that are being processed by a server event sink for local delivery.
Local Queue Length. This is the current count of messages in the local queue.
Local Retry Queue Length. This is the current count of messages in the local retry queue.
Messages Pending Routing. This is the current count of messages that have been categorized but not routed.
Remote Queue Length. This is the current count of messages in the remote queue.
Remote Retry Queue Length. This is the current count of messages in the retry queue for remote delivery.

Monitoring message system usage

The counters listed in this section are those that provide an indicator as to how heavily the e-mail system is being used. Unlike the first two groups of counters, these do not need to be as closely watched. Instead, the administrator should periodically collect data for these counters and compare the collected data to the system baseline created when the e-mail system was first configured. The need to add additional hardware resources often can be justified by the e-mail usage increases shown with these counters.

MSExchangeIS

Active User Count. This is the number of user connections that have shown some activity in the last 10 minutes.
User Count. User Count is the number of users connected to the Information Store.

MSExchangeIS Mailbox

Messages Delivered/min. This counter tracks the rate at which messages are being delivered to the Private Information Store. This includes both messages delivered to the Information Store by the MTA and those submitted directly to the Information Store from clients on this server.

This counter should usually be in the range of 10 to 40 messages per minute. If the value is constantly less than five messages per minute while there are pending items in the MTA queue, then it is possible that the server is under a heavy load or there is a problem with one of the processes. If this value is very high (e.g., greater than 200 messages per minute) for an extended period, there may be a message stuck in the MTA queue.
Messages Sent/min. This counter tracks the rate at which messages are sent from the Information Store to the MTA to be transported to other servers or gateways.

MSExchangeIS Public

Messages Delivered/min. This counter tracks the rate at which messages are being delivered to the Public Information Store. This includes both messages delivered to the Information Store by the MTA and those submitted directly to the Information Store from clients on this server.
Messages Sent/min. This counter tracks the rate at which messages are sent from the Public Information Store to the MTA to be transported to other servers or gateways.

MSExchangeMTA

LAN Receive Bytes/sec. This is the rate at which bytes are received over a LAN from MTAs.
LAN Transmit Bytes/sec. This is the rate at which bytes are transmitted over a LAN to MTAs.
Messages/Sec. This counter is the running average of the rate that messages are processed. This counter can be used to monitor the message traffic between servers.

MSExchangeNMC

If you are running a Notes Connector, you should monitor the following counters:

Rate of messages received. This is the rate that Exchange is receiving Lotus Notes messages.
Rate of messages sent. This is the rate that Exchange messages are being sent to Lotus Notes.

SMTP Server

Messages Delivered/sec. This is the rate at which messages are delivered to local mailboxes.
Messages Received/sec. This is the rate at which inbound messages are being received.
Messages Sent/sec. This is the rate at which outbound messages are being sent.

Counters for troubleshooting

The primary role of the remaining performance monitor counters is to aid with troubleshooting. There are many Exchange-related objects, including:

MSADC
MSExchange Oledb Events
MSExchange Oledb Resource
MSExchange Web Mail
MSExchangeActiveSyncNotify OmaPush
MSExchangeAL
MSExchangeDSAccess Caches
MSExchangeDSAccess Domain Controllers
MSExchangeDSAccess Global Counters
MSExchangeDSAccess Processes
MSExchangeES
MSExchangeIS
MSExchangeIS Mailbox
MSExchangeIS Public
MSExchangeIS Transport Driver
MSExchangeMTA
MSExchangeMTA Connections
MSExchangeOMA
MSExchangeSA-NSPI Proxy
MSExchangeSRS
MSExchangeTransport Store Driver
SMTP NTFS Store Driver
SMTP Server

Hundreds of counters for these objects can be monitored as needed to diagnose problems. The following is a small sample of the additional objects that may be useful for troubleshooting.

MSExchangeIS Mailbox

Average Delivery Time. This counter shows the average length of time that the 10 most recent messages waited in the Information Store queue before being transferred to the MTA. A high value often indicates an MTA performance problem.
Message Recipients Delivered/min. This counter shows a continuous average of the number of messages sent per minute divided by the number of recipients to which the messages were sent. This provides a fairly accurate count of the actual number of deliveries.

MSExchangeIS Public

Average Delivery Time. This counter shows the average length of time that the 10 most recent messages waited in the Public Information Store queue before being transferred to the MTA. A high value often indicates an MTA performance problem.
Message Recipients Delivered/min. This counter shows a continuous average of the number of messages sent per minute divided by the number of recipients to which the messages were sent. This provides a fairly accurate count of the actual number of deliveries.

SMTP Server

Inbound Connections Current. This is the current count of connections to the SMTP service established by other SMTP hosts.
Inbound Connections Total. This is the total number of connections the SMTP service has accepted from other hosts since the service was started.
Outbound Connections Current. This is the current count of connections the SMTP service has established to other SMTP hosts.
Outbound Connections Refused. This is the total number of connections the SMTP service has attempted to other hosts that have been refused since the service was started.
Outbound Connections Total. This is the total count of successful connections that the SMTP service has established since the service was started.
Messages Received Total. This is the total number of inbound messages accepted.
NDRs Generated. This counter shows the total number of nondelivery reports generated for inbound mail.
Messages Sent Total. This is the total number of outbound messages delivered to their destinations.
Message Bytes Received Total. This counter shows the total size of all inbound messages transferred to Exchange.
Message Bytes Sent Total. This counter shows the total size of all outbound messages transferred from Exchange.

< Day Day Up >