Section 2.9. mpstat Command

2.9. mpstat Command

The mpstat command summarizes the utilization statistics for each CPU. The following output is an example from a system with 32 CPUs.

$ mpstat 1 CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl   0    0   0  279   267  112  106    7    7   85    0   219   85   2   0  13   1    1   0  102    99    0  177    9   15  119    2   381   72   3   0  26   2    2   0   75   130    0  238   19   11   98    5   226   70   3   0  28   3    1   0   94    32    0   39    8    6   95    2   380   81   2   0  17   4    1   0   70    75    0  128   11    9   96    1   303   66   2   0  32   5    1   0   88    62    0   99    7   11   89    1   370   74   2   0  24   6    4   0   78    47    0   85   24    6   67    8   260   86   2   0  12   7    2   0   73    29    0   45   21    5   57    7   241   85   1   0  14   8    2   0   94    21    0   17   10    3   73    0   392   86   2   0  12   9    3   0   64   106    0  198   23   12   85    7   209   66   2   0  32  10    2   0   23   164    0  331   20   13   85    7    89   17   2   0  81  11    3   0   31   105    0  200   14   11   63    4   144   33   2   0  66  12    0   0   47   129    0  260    3    8   86    0   248   33   2   0  65  13    4   0   76    48    0   90   25    5   77    8   255   75   2   0  24  14    5   0   39   130    0  275   17    9   83   10   158   36   2   0  62  15    2   0   67    99    0  183   18    5  101    4   207   72   2   0  25  16    3   0   79    61    0  103   20    2   89    3   252   83   2   0  15  17    2   0   17    64    0  123    8    7   38    2    65   18   1   0  81  18    4   0   86    22    0   21   21    0   98    4   283   98   2   0   0  19    1   0   10    75    0  148    9    5   38    1    47    9   3   0  88  20    1   0   49   108    1  208    4    6   85    0   252   29   2   0  69  21    5   0   90    46    0   77   29    1   75    8   277   92   2   0   6  22    2   0   56    98    0  186   15    5   71    3   176   59   2   0  39  23    5   0   37   156    0  309   19    6   75    4   136   39   2   0  59  24    0   0   32    51    0   97    2    3   32    1   198   15   1   0  83  25    8   0   82    56    0  142   13    8   87   13   294   82   2   0  16  26    2   0   66    40    0   75   12    3   66    2   237   73   2   0  25  27    6   0   80    33    0   57   21    5   89    3   272   86   2   0  13  28    1   0   97    35    0   56    7    3   94    2   369   76   2   0  22  29    4   0   83    44    0   79   27    3   69    7   286   89   2   0   9  30    1   0   95    41    1   69    8    4  105    1   382   80   2   0  18  31    5   0   16    31    2   99    5    9   20   15    97    9   1   0  90 ... 

For each CPU, a set of event counts and utilization statistics are reported. The first output printed is the summary since boot. After vmstat is checked, the mpstat processor utilization metrics are often the next point of call to ascertain how busy the system CPUs are.

Processor utilization is reported by percent user (usr), system (sys), wait I/O (wt) and idle (idl) times, which have the same meanings as the equivalent columns from vmstat (Section 2.2) and sar (Section 2.7). The syscl field provides additional information for understanding why system time was consumed.

• syscl(system calls). System calls per second. See Section 2.13 for an example of how to use DTrace to investigate the impact and cause of system call activity.

The scheduling-related statistics reported by mpstat are as follows.

• csw (context switches). This field is the total of voluntary and involuntary context switches. Voluntary context switches occur when a thread performs a blocking system call, usually one performing I/O when the thread voluntarily sleeps until the I/O event has completed.

• icsw (number of involuntary context switches). This field displays the number of threads involuntarily taken off the CPU either through expiration of their quantum or preemption by a higher-priority thread. This number often indicates if there were generally more threads ready to run than physical processors. To analyze further, a DTrace probe, dequeue, fires when context switches are made, as described in Section 2.15.

• migr(migrations of threads between processors). This field displays the number of times the OS scheduler moves ready-to-run threads to an idle processor. If possible, the OS tries to keep the threads on the last processor on which it ran. If that processor is busy, the thread migrates. Migrations on traditional CPUs are bad for performance because they cause a thread to pull its working set into cold caches, often at the expense of other threads.

• intr (interrupts). This field indicates the number of interrupts taken on the CPU. These may be hardware- or software-initiated interrupts. See Section 3.11 in Solaris^™ Internals for further information.

• ithr (interrupts as threads). The number of interrupts that are converted to real threads, typically as a result of inbound network packets, blocking for a mutex, or a synchronization event. (High-priority interrupts won't do this, and interrupts without mutex contention typically interrupt the running thread and complete without converting to a full thread). See Section 3.11 in Solaris^™ Internals for further information.

The locking-related statistics reported by mpstat are as follows.

• smtx (kernel mutexes). This field indicates the number of mutex contention events in the kernel. Mutex contention typically manifests itself first as system time (due to busy spins), which results in high system (%sys) time, which don't show in smtx. More useful lock statistics are available through lockstat(1M) and the DTrace lockstat provider (see Section 9.3.5 and Chapter 17 in Solaris^™ Internals).

• srw (kernel reader/writer mutexes). This field indicates the number of reader-writer lock contention events in the kernel. Excessive reader/writer lock contention typically results in nonscaling performance and systems that are unable to use all the available CPU resources (symptom is idle time). More useful lock statistics are available with lockstat(1M) and the DTrace lockstat providerSee Section 9.3.5 and Chapter 17 in Solaris^™ Internals.

See Chapter 3 in Solaris^™ Internals, particularly Section 3.8.1, for further information.