sar


sar is a great general performance monitoring tool. sar can output data for almost everything Linux does. The sar command is delivered in the sysstat rpm. We use sysstat version 5.0.5 in our examples. This is one of the more recent versions listed as stable. Look at the sysstat home page at http://perso.wanadoo.fr/sebastien.godard/ for release information and downloads.

sar can display performance data for CPU, run queue, disk I/O, paging (swap), memory, CPU interrupts, networking, and more. The most important sar feature is the capability to create a data file. Every Linux system should collect sar data from cron jobs. The sar data file provides a system administrator with historical performance information. This feature is very important, and it separates sar from the other performance tools. If a nightly batch job runs twice as long as normal, you won't find out until the next morning (unless you get paged). You need the ability to look at performance data from 12 hours ago. The sar data collector provides this ability. Many reporting syntaxes exist, but let's look at data collection first.

sar Data Collector

sar data collection is done with a binary executable and two scripts in /usr/lib/sa. The sar data collector is a binary executable located at /usr/lib/sa/sadc. The job of sadc is to write to the data collection file /var/log/sa/. Several options can be supplied to sadc. A common syntax is:

/usr/lib/sa/sadc interval iterations file name

interval is the number of seconds between sampling. iterations is the number of samples to take. file name specifies the output file. A simple sadc syntax is /usr/lib/ sa/sadc 360 5 /tmp/sadc.out. This command takes five samples at five-minute intervals and stores them in /tmp/sadc.out. We should collect the samples on a regular basis, so we need a script to be run by cron. We should put the samples in a place that makes sense, like we did with the top script from the previous section. Fortunately, the sysstat rpm provides the /usr/lib/sa/sa1 script to do all this.

The sa1(8) man page is much longer than the sa1 script itself. /usr/lib/sa/sa1 is a very simple script that runs sadc with the syntax sadc -F -L 1 1 /var/log/sa/sa## where ## is the day of the month. Older versions of sa1 use the output from date +.%Y_%m_%d as the file suffix. The -F option causes sadc to force creation of the output file if necessary. The -L locks the output file before writing to it to prevent corrupting the file when two sadc processes are running at the same time. Older versions of sadc didn't have the -L option, so the sa1 script performed manual locking. The only options for the sa1 script are the interval between samples and the number of iterations to sample. A cron file (/etc/cron.d/sysstat) is supplied with sysstat. It differs between sysstat versions. The following are the entries for the version 5.0.5 of sysstat:

# cat /etc/cron.d/sysstat # run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib/sa/sa1 1 1 # generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib/sa/sa2 -A


You can see that after the sysstat rpm is installed, sadc begins taking samples. The sysstat home page is http://perso.wanadoo.fr/sebastien.godard/.[2] The documentation link offers the following crontab suggestions as of January 14, 2006:

# 8am-7pm activity reports every 10 minutes during weekdays. 0 8-18 * * 1-5 /usr/lib/sa/sa1 600 6 & # 7pm-8am activity reports every an hour during weekdays. 0 19-7 * * 1-5 /usr/lib/sa/sa1 & # Activity reports every an hour on Saturday and Sunday. 0 * * * 0,6 /usr/lib/sa/sa1 & # Daily summary prepared at 19:05 5 19 * * * /usr/lib/sa/sa2 -A &


The crontab example in Sebastien Godard's Web site suggests taking a sample every 10 minutes from 8 a.m. to 6 p.m. weekdays and every hour otherwise. (Note: The crontab comment says 7pm but 18:00 is 6pm.) If disk space in /var is sufficient, you may want to sample every 10 minutes every hour of the day. If the weekend backups are slower, then an hourly sadc sample may not be very helpful.

Now let's look at the more popular reporting syntaxes.

CPU Statistics

The sar -u output shows CPU information. The -u option is the default for sar. The output shows CPU utilization as a percentage. Table 3-2 explains the output.

Table 3-2. sar -u Fields

Field

Description

CPU

CPU number

%user

Time spent running processes in user mode

%nice

Time spent running niced processes

%system

Time spent running processes in kernel mode (system)

%iowait

Time during which the processor was waiting for I/O to complete while no process was executing on the CPU

%idle

Time during which no process was executing on the CPU


This should look familiar. It is the same CPU information as in top reports. The following shows the output format:

[root@fisher dave]# sar 5 10 Linux 2.4.21-27.EL (fisher)     04/30/2005 02:03:20 PM       CPU     %user     %nice   %system   %iowait      %idle 02:03:25 PM       all     36.80      0.00      4.20      0.00      59.00 02:03:30 PM       all     37.80      0.00      5.20      2.20      54.80 02:03:35 PM       all     55.40      0.00      4.40      3.00      37.20 02:03:40 PM       all     53.60      0.00      6.20      0.00      40.20 02:03:45 PM       all     37.20      0.00      6.60      1.00      55.20 02:03:50 PM       all     36.60      0.00      4.40      2.20      56.80 02:03:55 PM       all     51.00      0.00      4.20      1.00      43.80 02:04:00 PM       all     55.60      0.00      4.20      0.00      40.20 02:04:05 PM       all     40.60      0.00      8.00      1.20      50.20 02:04:10 PM       all     36.60      0.00      4.20      2.20      57.00 Average:          all     44.12      0.00      5.16      1.28      49.44


The 5 10 causes sar to take 10 samples at 5-second intervals. The first column of any sar report is a timestamp.

We could have looked at the file created with sadc by using the -f option. This sar syntax shows the output from sar -f /var/log/sa/sa21:

[dave@fisher dave]$ sar -f /var/log/sa/sa21head -n 20 Linux 2.4.21-20.EL (fisher)   04/21/2005 12:00:00 AM       CPU     %user     %nice   %system   %iowait      %idle 12:10:00 AM       all      0.23      0.00      0.23      0.02      99.52 12:20:00 AM       all      0.22      0.00      0.20      0.01      99.57 12:30:01 AM       all      0.21      0.00      0.19      0.01      99.59 12:40:00 AM       all      0.23      0.00      0.22      0.02      99.54 12:50:01 AM       all      0.19      0.00      0.28      0.01      99.52 01:00:00 AM       all      0.22      0.00      0.18      0.01      99.59 01:10:00 AM       all      0.40      0.00      0.25      0.02      99.34 01:20:00 AM       all      0.20      0.00      0.25      0.01      99.53 01:30:00 AM       all      0.20      0.00      0.23      0.01      99.56 01:40:00 AM       all      0.22      0.00      0.21      0.02      99.56 01:50:00 AM       all      0.22      0.00      0.20      0.02      99.56 02:00:00 AM       all      0.22      0.00      0.19      0.01      99.58 02:10:00 AM       all      0.23      0.00      0.24      0.02      99.50 02:20:01 AM       all      0.22      0.00      0.25      0.01      99.52 02:30:00 AM       all      0.19      0.00      0.22      0.01      99.57 02:40:00 AM       all      0.22      0.00      0.22      0.01      99.55 02:50:00 AM       all      0.21      0.00      0.21      0.01      99.56


The sar command can break down this information for each CPU in a multi-CPU Linux box too, as the following sar -u -P ALL 5 5 output demonstrates:

Linux 2.4.21-20.ELsmp (doughboy)      04/13/2005 10:17:56 AM       CPU     %user     %nice   %system   %iowait      %idle 10:18:01 AM       all     26.41      0.00     23.46      0.00      50.13 10:18:01 AM         0     23.20      0.00     32.80      0.00      44.00 10:18:01 AM         1     26.60      0.00     20.80      0.00      52.60 10:18:01 AM         2     25.80      0.00     21.00      0.00      53.20 10:18:01 AM         3     30.06      0.00     19.24      0.00      50.70 10:18:01 AM       CPU     %user     %nice   %system   %iowait      %idle 10:18:06 AM       all     17.70      0.00     24.15      0.00      58.15 10:18:06 AM         0     22.40      0.00     26.40      0.00      51.20 10:18:06 AM         1     15.20      0.00     24.60      0.00      60.20 10:18:06 AM         2     19.00      0.00     20.00      0.00      61.00 10:18:06 AM         3     14.20      0.00     25.60      0.00      60.20 10:18:06 AM       CPU     %user     %nice   %system   %iowait      %idle 10:18:11 AM       all     13.69      0.00     23.74      0.05      62.52 10:18:11 AM         0      9.00      0.00     27.40      0.00      63.60 10:18:11 AM         1     19.40      0.00     20.40      0.20      60.00 10:18:11 AM         2     13.20      0.00     21.00      0.00      65.80 10:18:11 AM         3     13.17      0.00     26.15      0.00      60.68 10:18:11 AM       CPU     %user     %nice   %system   %iowait      %idle 10:18:16 AM       all     16.40      0.00     23.00      0.00      60.60 10:18:16 AM         0     16.60      0.00     18.00      0.00      65.40 10:18:16 AM         1     15.00      0.00     23.00      0.00      62.00 10:18:16 AM         2     19.40      0.00     19.80      0.00      60.80 10:18:16 AM         3     14.60      0.00     31.20      0.00      54.20 10:18:16 AM       CPU     %user     %nice   %system   %iowait      %idle 10:18:21 AM       all     32.60      0.00     22.10      0.00      45.30 10:18:21 AM         0     30.80      0.00     24.40      0.00      44.80 10:18:21 AM         1     34.80      0.00     24.00      0.00      41.20 10:18:21 AM         2     32.00      0.00     20.20      0.00      47.80 10:18:21 AM         3     32.80      0.00     19.80      0.00      47.40 Average:          CPU     %user     %nice   %system   %iowait      %idle Average:          all     21.36      0.00     23.29      0.01      55.34 Average:            0     20.40      0.00     25.80      0.00      53.80 Average:            1     22.20      0.00     22.56      0.04      55.20 Average:            2     21.88      0.00     20.40      0.00      57.72 Average:            3     20.96      0.00     24.40      0.00      54.64


Disk I/O Statistics

sar is a good tool for looking at disk I/O. The following shows a sample of sar disk I/O output.

[dave@fisher dave]$ sar -d 5 2 Linux 2.4.21-27.EL (fisher)     04/29/2005 04:22:03 PM       DEV       tps  rd_sec/s   wr_sec/s 04:22:08 PM    dev3-0      1.40      0.00     784.00 04:22:08 PM    dev3-1      0.00      0.00       0.00 04:22:08 PM    dev3-2      1.40      0.00     784.00 04:22:08 PM    dev3-3      0.00      0.00       0.00 04:22:08 PM   dev3-64      0.00      0.00       0.00 04:22:08 PM   dev3-65      0.00      0.00       0.00 04:22:08 PM       DEV       tps  rd_sec/s   wr_sec/s 04:22:13 PM    dev3-0     34.60      0.00    4219.20 04:22:13 PM    dev3-1      0.00      0.00       0.00 04:22:13 PM    dev3-2     34.60      0.00    4219.20 04:22:13 PM    dev3-3      0.00      0.00       0.00 04:22:13 PM   dev3-64      0.00      0.00       0.00 04:22:13 PM   dev3-65      0.00      0.00       0.00 Average:          DEV       tps  rd_sec/s   wr_sec/s Average:       dev3-0     18.00      0.00    2501.60 Average:       dev3-1      0.00      0.00       0.00 Average:       dev3-2     18.00      0.00    2501.60 Average:       dev3-3      0.00      0.00       0.00 Average:      dev3-64      0.00      0.00       0.00 Average:      dev3-65      0.00      0.00       0.00


The -d shows disk I/O information. The 5 2 options are interval and iterations, just like the sar data collector. Table 3-3 lists the fields and descriptions.

Table 3-3. sar -d Fields

Field

Description

DEV

Disk device

tps

Transfers per second (or IOs per second)

rd_sec/s

512-byte reads per second

wr_sec/s

512-byte writes per second


512 is just a unit of measure. It doesn't imply that all disk I/O is in 512-byte chunks. The DEV column is the disk device in the format dev#-#, where the first # is a device major number, and the second # is a minor or sequential number. sar uses the minor number with kernels greater than 2.5. For example, we saw dev3-0 and dev3-1 in the sar -d output. These correspond to /dev/hda and /dev/hda1. Look at the following entries in /dev:

brw-rw----    1 root     disk       3,   0 Jun 24  2004 hda brw-rw----    1 root     disk       3,   1 Jun 24  2004 hda1


/dev/hda has major number 3 and minor number 0. hda1 has a major number of 3 and a minor number of 1.

Networking Statistics

sar offers four different syntax options to display networking information. The -n option takes four different switches: DEV, EDEV, SOCK, and FULL. DEV displays networking interface information, EDEV shows statistics about network errors, SOCK shows socket information, and FULL shows all three switches. They can be used separately or together. Table 3-4 shows the fields reported with the -n DEV option.

Table 3-4. sar -n DEV Fields

Field

Description

IFACE

LAN interface

rxpck/s

Packets received per second

txpck/s

Packets transmitted per second

rxbyt/s

Bytes received per second

txbyt/s

Bytes transmitted per second

rxcmp/s

Compressed packets received per second

txcmp/s

Compressed packets transmitted per second

rxmcst/s

Multicast packets received per second


The following is the sar output using the -n DEV option:

# sar -n DEV 5 3 Linux 2.4.21-20.EL (fisher)     04/20/2005 03:39:18 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s txcmp/ s  rxmcst/s 03:39:23 AM        lo      0.00      0.00      0.00      0.00      0.00 0.0 0      0.00 03:39:23 AM      eth0     36.80      0.00   8412.20     10.80      0.00 0.0 0      0.00 03:39:23 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s txcmp/ s  rxmcst/s 03:39:28 AM        lo      0.00      0.00      0.00      0.00       0.00 0.0 0      0.00 03:39:28 AM      eth0     50.20      1.80   9798.20    526.60       0.00 0.0 0      0.00 03:39:28 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s txcmp/ s  rxmcst/s 03:39:33 AM        lo      0.00      0.00      0.00      0.00       0.00 0.0 0      0.00 03:39:33 AM      eth0     38.20      1.80   8882.60    400.80       0.00 0.0 0      0.00 Average:        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s txcmp/ s  rxmcst/s Average:           lo      0.00      0.00      0.00      0.00       0.00 0.0 0      0.00 Average:         eth0     41.73      1.20   9031.00    312.73       0.00 0.0 0      0.00


Information about networking errors can be displayed with sar -n EDEV. Table 3-5 lists the fields displayed.

# sar -n EDEV 5 3 Linux 2.4.21-20.EL (fisher)     04/17/2005 10:41:44 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s txcarr/s  rxfram/s  rxfifo/s  txfifo/s 10:41:49 AM        lo      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 10:41:49 AM      eth0      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 10:41:49 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s txcarr/s  rxfram/s  rxfifo/s  txfifo/s 10:41:54 AM        lo      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 10:41:54 AM      eth0      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 10:41:54 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s txcarr/s  rxfram/s  rxfifo/s  txfifo/s 10:41:59 AM        lo      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 10:41:59 AM      eth0      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 Average:        IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s txcarr/s  rxfram/s  rxfifo/s  txfifo/s Average:           lo      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00 Average:         eth0      0.00      0.00      0.00      0.00       0.00 0.00      0.00      0.00      0.00


Table 3-5. sar -n EDEV Fields

Field

Description

IFACE

LAN interface

rxerr/s

Bad packets received per second

txerr/s

Bad packets transmitted per second

coll/s

Collisions per second

rxdrop/s

Received packets dropped per second because buffers were full

txdrop/s

Transmitted packets dropped per second because buffers were full

txcarr/s

Carrier errors per second while transmitting packets

rxfram/s

Frame alignment errors on received packets per second

rxfifo/s

FIFO overrun errors per second on received packets

txfifo/s

FIFO overrun errors per second on transmitted packets


The SOCK argument displays IPCS socket information. Table 3-6 lists the fields displayed and their meanings.

# sar -n SOCK 5 3 Linux 2.4.21-144-default (sawnee)       04/17/05 16:00:56       totsck    tcpsck    udpsck    rawsck    ip-frag 16:01:01          117        11         8         0          0 16:01:06          117        11         8         0          0 16:01:11          117        11         8         0          0 Average:          117        11         8         0          0


Table 3-6. sar -n SOCK Fields

Field

Description

totsck

Total number of sockets used

tcpsck

Number of TCP sockets used

udpsck

Number of UDP sockets used

rawsck

Number of raw sockets used

ip-frag

Number of IP fragments used


sar can generate many other reports. It is worth reading through the sar(1) man page to see whether there are any others you want to start using.



Linux Troubleshooting for System Administrators and Power Users
Real World Mac Maintenance and Backups
ISBN: 131855158
EAN: 2147483647
Year: 2004
Pages: 129
Authors: Joe Kissell

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net