Detecting and Resolving Performance Issues Through Monitoring | MicrosoftВ® WindowsВ® Command-Line Administrators Pocket Consultant

< Day Day Up >

Although Pmon provides an excellent starting point for detecting and resolving performance issues, you’ll often need to dig deeper to determine whether a problem exists and if so, what is the possible cause. Several process management and monitoring tools can help you perform detailed analysis, including

Memory Monitor (Memmonitor) Displays detailed information regarding the memory a process is using.
Page Fault Monitor (Pfmon) Displays detailed information regarding page faults that are occurring on the system.
Resource Leak Triage Tool (Memtriage) Logs memory usage, including detailed information regarding memory allocated and freed by individual processes. Used to detect memory leaks and get memory pool details.

Note

Memmonitor, Pfmon, and Memtriage are intended for use by administrators. They are available in the Windows Server 2003 Resource Kit. If they wish, administrators can install the Windows Server 2003 Resource Kit on Windows XP Professional systems, and then the tools would be available on those systems.

These tools are discussed in the sections that follow.

Monitoring Memory Paging for Individual Processes

You can use the Page Fault Monitor (Pfmon) to get detailed data related to hard and soft page faults.

To see soft page faults, type

pfmon /c /p ProcessID

where ProcessID is the identification number of the process you want to monitor as determined by Tasklist, such as

pfmon /c /p 1348

To see hard page faults, type

pfmon /h /p ProcessID

where ProcessID is the identification number of the process you want to monitor as determined by Tasklist, such as

pfmon /h /p 1348

The output of Pfmon shows the page faults as they occur in real time. To stop Pfmon, press Ctrl+C. Page faults are shown according to the source that generated them and are similar to the following example:

HARD: HttpSendRequestExA+0xab1 : 00878000
HARD: URLQualifyW+0x356e : URLQualifyW+0x0000356D
HARD: CreateDataCache+0x3608 : 
WdtpInterfacePointer_UserFree+0x00007165
HARD: OleSetClipboard+0x8df : OleSetClipboard+0x000008DE
HARD: DoFileDownloadEx+0xba2 : DoFileDownloadEx+0x00000BA1
HARD: RtlImageNtHeaderEx+0x3d : 03730000
HARD: RtlSetThreadErrorMode+0x2a1 : 60c0100c
HARD: GetSysColorBrush+0xa4 : 60c0e5f8

Although developers will be interested in the source of page faults, administrators are more interested in how many page faults are occurring. As discussed earlier, most processors can handle large numbers of soft faults. A soft fault simply means the system had to look elsewhere in memory for the requested memory page. With a hard fault, on the other hand, the requested memory page must be retrieved from disk and if there are a lot of hard faults, you may need to increase the amount of memory or reduce the system cache size.

Monitoring Memory Usage and the Working Memory Set for Individual Processes

You use the Memory Monitor (Memmonitor) to track detailed memory usage for individual processes. The syntax is

memmonitor /p ProcessID /nodbg [/int IntervalSeconds]

where ProcessID is the identification number of the process you want to monitor as determined by Tasklist, /nodbg tells the utility not to break to the debugger when finished, and IntervalSeconds sets the optional number of seconds to wait between each memory usage check. The default wait interval is 60 seconds.

Memmonitor output looks like this:

Monitor Process 1284 (Name: SQLAgent.exe)MemMon - 0:00:00
    PageFaults       : 13182
    PeakWSSize       : 22704K     WorkingSetSize: 22252K
    PeakPagedPool    :    58K     PagedPool     :    54K
    PeakNonPagedPool :     8K     NonPagedPool  :     7K
    PeakPagefile     : 13632K     Pagefile      : 13176K
MemMon - 0:00:30
    PageFaults       : 16259
    PeakWSSize       : 24800K     WorkingSetSize: 24352K
    PeakPagedPool    :    58K     PagedPool     :    54K
    PeakNonPagedPool :     8K     NonPagedPool  :     8K
    PeakPagefile     : 16256K     Pagefile      : 15804K

As you can see from the example, Memmonitor shows the elapsed running time, following by the current memory usage details. As with Pfmon, Memmonitor runs continuously. However, it displays output at the given interval only when there are memory usage changes for the specified process. You can stop Memmonitor at any time by pressing Ctrl+C.

The Memmonitor fields provide the following information:

PageFaults Shows the number of hard and soft page faults that have occurred for the process
PeakWSSize Shows the peak amount of memory used by the process
PeakPagedPool Shows the peak amount of paged memory used by the process
PeakNonPagedPool Shows the peak amount of nonpaged memory used by the process
PeakPagefile Shows the peak amount of page file memory used by the process
WorkingSetSize Shows the amount of memory allocated to the process by the operating system
PagedPool Shows the amount of allocated memory that is allowed to be paged to the hard disk
NonPagedPool Shows the amount of allocated memory that can’t be written to disk
Pagefile Shows the size of the file on the hard disk to which memory may be paged

When you work with Memmonitor, you are zeroing in on the memory usage of a specific process. The key things to monitor are memory paging and the working memory set. Ideally, if the memory cache for the process (and its related application) is allocated properly, the relative frequency of memory paging won’t be excessive. If paging does become excessive, you may need to allocate more memory-resident file cache for the application or add physical memory to the system.

Real World

Remember, this is all relative to the application being considered. Some applications will always have a lot of paging. Just keep in mind that accessing the page file is much slower than accessing physical memory.

The working set of memory shows how much memory is allocated to the process by the operating system. If the working set increases over time and doesn’t eventually go back to baseline usage, the process may have a memory leak. With a memory leak, the process isn’t properly releasing memory that it’s using and this can lead to reduced performance of the entire system.

Tip

If you suspect a memory leak, you’ll find several tools that can help you further analyze the problem, including the Resource Leak Triage Tool (Memtriage), Memory Snapshot (Memsnap) and Pool Monitor (Poolmon). In most cases, the best tool to use is Memtriage, which is discussed in the next section. Memtriage is available in the Windows Server 2003 Resource Kit, and Memsnap and Poolman are included in the Support Tools for both Windows XP Professional and Windows Server 2003.

Performing Detailed Memory Usage Analysis and Determining the Source of Memory Leaks

You use Memtriage to help you pinpoint the source of suspected memory leaks. With Memtriage, you use the following syntax:

memtriage /mp logName /t numSnapshots /w snapshotWindow

where /Mp tells Windows to take a snapshot of current system, process, and kernel pool information, logName is the name of the log file to use, numSnapshots is the number of times to write memory usage to the log, and snapshotWindow is the time interval between snapshots.

Tip

You don’t have to log system, process, and kernel pool information at the same time. Instead of using /Mp, you can use /M to log system and process information only, or /P to log only kernel pool information.

In most cases, you’ll want to create multiple snapshots and gather information over several hours. To see how this would work, consider the following example:

memtriage /mp C:\logs\memlog.log /t 8 /w 20

Here, eight memory snapshots are taken with 20 minutes between each snapshot and the logs are written to C:\logs\memlog.log: A partial output for Memtriage would be similar to the following:

Taking snapshot 1 @ 2003/10/24 19:32:32(Pacific Standard Time)
Creating local pool tag file: c:\localtag.txt ...
Poolsnap: Scan local pool tag file: c:\localtag.txt
Poolsnap: Scan pool tag file: C:\Program Files\Windows Resource 
Kits\Tools\pooltag.txt
Sleeping 20 minutes
Taking snapshot 2 @ 2003/10/24 19:52:32(Pacific Standard Time)
Creating local pool tag file: c:\localtag.txt ...
Poolsnap: Scan local pool tag file: c:\localtag.txt
Poolsnap: Scan pool tag file: C:\Program Files\Windows Resource 
Kits\Tools\pooltag.txt

As you can see, Memtriage creates a working file called Localtag.txt and examines a resource kit file called Pooltag.txt. Using the /Mp option with a log name of Mem.log, it also writes three log files:

Mem.log.system Contains the snapshot(s) of system information
Mem.log.process Contains the snapshot(s) of process information
Mem.log.pool Contains the snapshot(s) of kernel pool information

Note

Don’t worry—you don’t have to dig through all these log files. You should remember, however, that the log files are there, and use Memtriage to clean up those with which you have finished working.

When Memtriage finishes, you’ll want to analyze the logs you’ve made, looking for potential memory leaks. To do this, use the following syntax:

memtriage /a logName

where logName is the log specified in the /Mp option, such as

memtriage /a c:\logs\memlog.log

The output of the memory triage analysis is very detailed. Not only does the analysis reveal changes in memory usage; it also points out usage trends and the rate of change on an hourly basis. An example of the analysis output follows:

===================================  System  =====================================
Name          Inc-Trend Object         Change   Start     End  Percent  Rate/hour
System        Always    AvailableKByte  -1204  261928  260724        0     -21572
System        Sometime  NpagedPoolKByte    16    8276    8292        0        286
 
==================================  Per Process  =================================
Name          Inc-Trend Object         Change   Start     End  Percent  Rate/hour
sqlgea.exe    Sometime  CommitKByte        32    7536    7568        0        573
sqlgea.exe    Sometime  VirtualKByte      256   47956   48212        0       4586
sqlgea.exe    Sometime  Handles             7     533     540        1        125
sqlgea.exe    Sometime  Threads             1      20      21        5         17
sqlmon.exe    Sometime  PagedPoolKByte      1      77      78        1         17
sqlmon.exe    Sometime  NpagedPoolKByte     1      85      86        1         17
sqlmon.exe    Sometime  CommitKByte        52   29604   29656        0        931
sqlmon.exe    Sometime  VirtualKByte     1040   94280   95320        1      18634
sqlmon.exe    Sometime  Handles            10     994    1004        1        179
sdman.exe     Sometime  Handles             1     141     142        0         17
sdman.exe     Always    Handles             4     868     872        0         71
rrsrvc.exe    Sometime  Handles             2      83      85        2         35
sqldbms.exe   Sometime  Handles             2    2050    2052        0         35
mret.exe      Sometime  CommitKByte        12   13320   13332        0        215
mret.exe      Sometime  VirtualKByte      512  353088  353600        0       9173
mret.exe      Sometime  Threads             1      45      46        2         17
wmiprvse.exe  Sometime  CommitKByte       216    2944    3160        7       3870
wmiprvse.exe  Sometime  VirtualKByte      768   29484   30252        2      13760

In this data, you would look at the following items:

Inc-Trend Indicates whether and the degree to which memory usage increases. Pay particular attention to processes whose memory usage always tends to increase.
Change, Start, End Denotes the initial memory usage logged, the final memory usage logged, and the change or difference between the two values. A negative change indicates that the process freed memory. A positive change indicates the process used additional memory.
Percent Specifies the percentage of change between the Start and End memory usage values. Watch for large percentage changes, but also note which processes show nominal increases. A nominal increase in successive analysis sets could point to a slow memory leak.
Rate/Hour Specifies the estimated rate at which memory usage would change on an hourly basis given the current change and the trend for change. Large changes can be indicators of a memory leak. However, the Rate/Hour values are only meaningful if you’ve based the analysis on multiple snapshots taken over a period of several hours.

< Day Day Up >