sar is a great general performance monitoring tool. sar can output data for almost everything Linux does. The sar command is delivered in the sysstat rpm. We use sysstat version 5.0.5 in our examples. This is one of the more recent versions listed as stable. Look at the sysstat home page at http://perso.wanadoo.fr/sebastien.godard/ for release information and downloads. sar can display performance data for CPU, run queue, disk I/O, paging (swap), memory, CPU interrupts, networking, and more. The most important sar feature is the capability to create a data file. Every Linux system should collect sar data from cron jobs. The sar data file provides a system administrator with historical performance information. This feature is very important, and it separates sar from the other performance tools. If a nightly batch job runs twice as long as normal, you won't find out until the next morning (unless you get paged). You need the ability to look at performance data from 12 hours ago. The sar data collector provides this ability. Many reporting syntaxes exist, but let's look at data collection first. sar Data Collectorsar data collection is done with a binary executable and two scripts in /usr/lib/sa. The sar data collector is a binary executable located at /usr/lib/sa/sadc. The job of sadc is to write to the data collection file /var/log/sa/. Several options can be supplied to sadc. A common syntax is: /usr/lib/sa/sadc interval iterations file name interval is the number of seconds between sampling. iterations is the number of samples to take. file name specifies the output file. A simple sadc syntax is /usr/lib/ sa/sadc 360 5 /tmp/sadc.out. This command takes five samples at five-minute intervals and stores them in /tmp/sadc.out. We should collect the samples on a regular basis, so we need a script to be run by cron. We should put the samples in a place that makes sense, like we did with the top script from the previous section. Fortunately, the sysstat rpm provides the /usr/lib/sa/sa1 script to do all this. The sa1(8) man page is much longer than the sa1 script itself. /usr/lib/sa/sa1 is a very simple script that runs sadc with the syntax sadc -F -L 1 1 /var/log/sa/sa## where ## is the day of the month. Older versions of sa1 use the output from date +.%Y_%m_%d as the file suffix. The -F option causes sadc to force creation of the output file if necessary. The -L locks the output file before writing to it to prevent corrupting the file when two sadc processes are running at the same time. Older versions of sadc didn't have the -L option, so the sa1 script performed manual locking. The only options for the sa1 script are the interval between samples and the number of iterations to sample. A cron file (/etc/cron.d/sysstat) is supplied with sysstat. It differs between sysstat versions. The following are the entries for the version 5.0.5 of sysstat: # cat /etc/cron.d/sysstat # run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib/sa/sa1 1 1 # generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib/sa/sa2 -A You can see that after the sysstat rpm is installed, sadc begins taking samples. The sysstat home page is http://perso.wanadoo.fr/sebastien.godard/.[2] The documentation link offers the following crontab suggestions as of January 14, 2006: # 8am-7pm activity reports every 10 minutes during weekdays. 0 8-18 * * 1-5 /usr/lib/sa/sa1 600 6 & # 7pm-8am activity reports every an hour during weekdays. 0 19-7 * * 1-5 /usr/lib/sa/sa1 & # Activity reports every an hour on Saturday and Sunday. 0 * * * 0,6 /usr/lib/sa/sa1 & # Daily summary prepared at 19:05 5 19 * * * /usr/lib/sa/sa2 -A & The crontab example in Sebastien Godard's Web site suggests taking a sample every 10 minutes from 8 a.m. to 6 p.m. weekdays and every hour otherwise. (Note: The crontab comment says 7pm but 18:00 is 6pm.) If disk space in /var is sufficient, you may want to sample every 10 minutes every hour of the day. If the weekend backups are slower, then an hourly sadc sample may not be very helpful. Now let's look at the more popular reporting syntaxes. CPU StatisticsThe sar -u output shows CPU information. The -u option is the default for sar. The output shows CPU utilization as a percentage. Table 3-2 explains the output.
This should look familiar. It is the same CPU information as in top reports. The following shows the output format: [root@fisher dave]# sar 5 10 Linux 2.4.21-27.EL (fisher) 04/30/2005 02:03:20 PM CPU %user %nice %system %iowait %idle 02:03:25 PM all 36.80 0.00 4.20 0.00 59.00 02:03:30 PM all 37.80 0.00 5.20 2.20 54.80 02:03:35 PM all 55.40 0.00 4.40 3.00 37.20 02:03:40 PM all 53.60 0.00 6.20 0.00 40.20 02:03:45 PM all 37.20 0.00 6.60 1.00 55.20 02:03:50 PM all 36.60 0.00 4.40 2.20 56.80 02:03:55 PM all 51.00 0.00 4.20 1.00 43.80 02:04:00 PM all 55.60 0.00 4.20 0.00 40.20 02:04:05 PM all 40.60 0.00 8.00 1.20 50.20 02:04:10 PM all 36.60 0.00 4.20 2.20 57.00 Average: all 44.12 0.00 5.16 1.28 49.44 The 5 10 causes sar to take 10 samples at 5-second intervals. The first column of any sar report is a timestamp. We could have looked at the file created with sadc by using the -f option. This sar syntax shows the output from sar -f /var/log/sa/sa21: [dave@fisher dave]$ sar -f /var/log/sa/sa21head -n 20 Linux 2.4.21-20.EL (fisher) 04/21/2005 12:00:00 AM CPU %user %nice %system %iowait %idle 12:10:00 AM all 0.23 0.00 0.23 0.02 99.52 12:20:00 AM all 0.22 0.00 0.20 0.01 99.57 12:30:01 AM all 0.21 0.00 0.19 0.01 99.59 12:40:00 AM all 0.23 0.00 0.22 0.02 99.54 12:50:01 AM all 0.19 0.00 0.28 0.01 99.52 01:00:00 AM all 0.22 0.00 0.18 0.01 99.59 01:10:00 AM all 0.40 0.00 0.25 0.02 99.34 01:20:00 AM all 0.20 0.00 0.25 0.01 99.53 01:30:00 AM all 0.20 0.00 0.23 0.01 99.56 01:40:00 AM all 0.22 0.00 0.21 0.02 99.56 01:50:00 AM all 0.22 0.00 0.20 0.02 99.56 02:00:00 AM all 0.22 0.00 0.19 0.01 99.58 02:10:00 AM all 0.23 0.00 0.24 0.02 99.50 02:20:01 AM all 0.22 0.00 0.25 0.01 99.52 02:30:00 AM all 0.19 0.00 0.22 0.01 99.57 02:40:00 AM all 0.22 0.00 0.22 0.01 99.55 02:50:00 AM all 0.21 0.00 0.21 0.01 99.56 The sar command can break down this information for each CPU in a multi-CPU Linux box too, as the following sar -u -P ALL 5 5 output demonstrates: Linux 2.4.21-20.ELsmp (doughboy) 04/13/2005 10:17:56 AM CPU %user %nice %system %iowait %idle 10:18:01 AM all 26.41 0.00 23.46 0.00 50.13 10:18:01 AM 0 23.20 0.00 32.80 0.00 44.00 10:18:01 AM 1 26.60 0.00 20.80 0.00 52.60 10:18:01 AM 2 25.80 0.00 21.00 0.00 53.20 10:18:01 AM 3 30.06 0.00 19.24 0.00 50.70 10:18:01 AM CPU %user %nice %system %iowait %idle 10:18:06 AM all 17.70 0.00 24.15 0.00 58.15 10:18:06 AM 0 22.40 0.00 26.40 0.00 51.20 10:18:06 AM 1 15.20 0.00 24.60 0.00 60.20 10:18:06 AM 2 19.00 0.00 20.00 0.00 61.00 10:18:06 AM 3 14.20 0.00 25.60 0.00 60.20 10:18:06 AM CPU %user %nice %system %iowait %idle 10:18:11 AM all 13.69 0.00 23.74 0.05 62.52 10:18:11 AM 0 9.00 0.00 27.40 0.00 63.60 10:18:11 AM 1 19.40 0.00 20.40 0.20 60.00 10:18:11 AM 2 13.20 0.00 21.00 0.00 65.80 10:18:11 AM 3 13.17 0.00 26.15 0.00 60.68 10:18:11 AM CPU %user %nice %system %iowait %idle 10:18:16 AM all 16.40 0.00 23.00 0.00 60.60 10:18:16 AM 0 16.60 0.00 18.00 0.00 65.40 10:18:16 AM 1 15.00 0.00 23.00 0.00 62.00 10:18:16 AM 2 19.40 0.00 19.80 0.00 60.80 10:18:16 AM 3 14.60 0.00 31.20 0.00 54.20 10:18:16 AM CPU %user %nice %system %iowait %idle 10:18:21 AM all 32.60 0.00 22.10 0.00 45.30 10:18:21 AM 0 30.80 0.00 24.40 0.00 44.80 10:18:21 AM 1 34.80 0.00 24.00 0.00 41.20 10:18:21 AM 2 32.00 0.00 20.20 0.00 47.80 10:18:21 AM 3 32.80 0.00 19.80 0.00 47.40 Average: CPU %user %nice %system %iowait %idle Average: all 21.36 0.00 23.29 0.01 55.34 Average: 0 20.40 0.00 25.80 0.00 53.80 Average: 1 22.20 0.00 22.56 0.04 55.20 Average: 2 21.88 0.00 20.40 0.00 57.72 Average: 3 20.96 0.00 24.40 0.00 54.64 Disk I/O Statisticssar is a good tool for looking at disk I/O. The following shows a sample of sar disk I/O output. [dave@fisher dave]$ sar -d 5 2 Linux 2.4.21-27.EL (fisher) 04/29/2005 04:22:03 PM DEV tps rd_sec/s wr_sec/s 04:22:08 PM dev3-0 1.40 0.00 784.00 04:22:08 PM dev3-1 0.00 0.00 0.00 04:22:08 PM dev3-2 1.40 0.00 784.00 04:22:08 PM dev3-3 0.00 0.00 0.00 04:22:08 PM dev3-64 0.00 0.00 0.00 04:22:08 PM dev3-65 0.00 0.00 0.00 04:22:08 PM DEV tps rd_sec/s wr_sec/s 04:22:13 PM dev3-0 34.60 0.00 4219.20 04:22:13 PM dev3-1 0.00 0.00 0.00 04:22:13 PM dev3-2 34.60 0.00 4219.20 04:22:13 PM dev3-3 0.00 0.00 0.00 04:22:13 PM dev3-64 0.00 0.00 0.00 04:22:13 PM dev3-65 0.00 0.00 0.00 Average: DEV tps rd_sec/s wr_sec/s Average: dev3-0 18.00 0.00 2501.60 Average: dev3-1 0.00 0.00 0.00 Average: dev3-2 18.00 0.00 2501.60 Average: dev3-3 0.00 0.00 0.00 Average: dev3-64 0.00 0.00 0.00 Average: dev3-65 0.00 0.00 0.00 The -d shows disk I/O information. The 5 2 options are interval and iterations, just like the sar data collector. Table 3-3 lists the fields and descriptions.
512 is just a unit of measure. It doesn't imply that all disk I/O is in 512-byte chunks. The DEV column is the disk device in the format dev#-#, where the first # is a device major number, and the second # is a minor or sequential number. sar uses the minor number with kernels greater than 2.5. For example, we saw dev3-0 and dev3-1 in the sar -d output. These correspond to /dev/hda and /dev/hda1. Look at the following entries in /dev: brw-rw---- 1 root disk 3, 0 Jun 24 2004 hda brw-rw---- 1 root disk 3, 1 Jun 24 2004 hda1 /dev/hda has major number 3 and minor number 0. hda1 has a major number of 3 and a minor number of 1. Networking Statisticssar offers four different syntax options to display networking information. The -n option takes four different switches: DEV, EDEV, SOCK, and FULL. DEV displays networking interface information, EDEV shows statistics about network errors, SOCK shows socket information, and FULL shows all three switches. They can be used separately or together. Table 3-4 shows the fields reported with the -n DEV option.
The following is the sar output using the -n DEV option: # sar -n DEV 5 3 Linux 2.4.21-20.EL (fisher) 04/20/2005 03:39:18 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:23 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:23 AM eth0 36.80 0.00 8412.20 10.80 0.00 0.0 0 0.00 03:39:23 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:28 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:28 AM eth0 50.20 1.80 9798.20 526.60 0.00 0.0 0 0.00 03:39:28 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:33 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:33 AM eth0 38.20 1.80 8882.60 400.80 0.00 0.0 0 0.00 Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s Average: lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 Average: eth0 41.73 1.20 9031.00 312.73 0.00 0.0 0 0.00 Information about networking errors can be displayed with sar -n EDEV. Table 3-5 lists the fields displayed. # sar -n EDEV 5 3 Linux 2.4.21-20.EL (fisher) 04/17/2005 10:41:44 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:49 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:49 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:49 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:54 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:54 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:54 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:59 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:59 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The SOCK argument displays IPCS socket information. Table 3-6 lists the fields displayed and their meanings. # sar -n SOCK 5 3 Linux 2.4.21-144-default (sawnee) 04/17/05 16:00:56 totsck tcpsck udpsck rawsck ip-frag 16:01:01 117 11 8 0 0 16:01:06 117 11 8 0 0 16:01:11 117 11 8 0 0 Average: 117 11 8 0 0
sar can generate many other reports. It is worth reading through the sar(1) man page to see whether there are any others you want to start using. |