Measuring IO | Performance Tuning for Linux Servers

Measuring I/O

The sysstat package contains the sar and iostat commands. The sar command collects and reports system activity information. The iostat command reports CPU utilization and I/O statistics for disks.

The following subsections discuss the iostat command in detail and describe how to use iostat and sar to analyze and report I/O and CPU utilization and allocation.

iostat

The iostat command monitors system input and output device loading by observing the time the devices are active in relation to their average transfer rates. The command generates reports that can be used to change the system configuration to better balance the input and output load between the physical disks.

As with most monitoring commands, the first line of iostat output reflects a summary of statistics since boot time. To look at meaningful real-time data, run iostat with a timestamp and look at the lines that report summaries over the time step intervals. iostat can provide a way to balance the load among the physical hard drives by viewing statistics from bytes read or written to the drive.

When bytes are either read or written, iostat reports the following information:

tps. Number of transfers per second.
device. Disk device name.
Blk_read/s. Amount of data read from the device, expressed in number of blocks per second.
Blk_wrtn/s. Amount of data written to the device, expressed in number of blocks per second.
Blk_read. Total number of blocks read.
Blk_wrtn. Total number of blocks written.
kB_read/s. Amount of data read from the device, expressed in kilobytes per second.
kB_wrtn/s. Amount of data written from the device, expressed in kilobytes per second.
kB_read. Total number of kilobytes read.
kB_wrtn. Total number of kilobytes written.
rrqm/s. Number of read requests merged per second that were issued to the device.
wrqm/s. Number of write requests merged per second that were issued to the device.
r/s. Number of read requests that were issued to the device per second.
w/s. Number of write requests that were issued to the device per second.
rsec/s. Number of sectors read from the device per second.
wsec/s. Number of sectors written from the device per second.
rkB/s. Number of kilobytes read from the device per second.
wkB/s. Number of kilobytes written to the device per second.
avgrp-sz. Average size (in sectors) of requests that were issued to the device.
avgqu-sz. Average queue length.

Using the iostat and sar Utilities

The Linux iostat and sar utilities analyze and report on I/O and CPU utilization and allocation by providing a simultaneous interval-by-interval profile of disk and CPU usage. These utilities can indicate which system resource might be limiting overall system performance if used during heavy workloads or periods of inadequate system performance. After the system bottleneck is identified, directed actions can be taken to improve system performance.

The following example walks through the execution of iostat on the /dev/hda1 device and presents the output.

[View full width]

# iostat -x /dev/hda1 Linux 2.4.20-4GB avg-cpu: %user %nice %sys %idle 0.67 0.00 0.18 99.15 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz

await svctm %util /dev/hda1 0.00 0.00 0.00 0.00 0.03 0.01 0.01 0.00 8.71 0.01

199.71 199.41 0.08

This example illustrates the execution of tiobench to produce file system activity on the /dev/hda1 device.

[View full width]

# date && ./tiobench.pl --size 500 --numruns 5\ -threads 32 && date Fri Jun 27 05:41:03 PDT 2003 Run #1: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T Run #2: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T Run #3: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T Run #4: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T Run #5: ./tiotest -t 32 -f 15 -r 125 -b 4096 -d . -T Unit information ================ File size = megabytes Blk Size = bytes Rate = megabytes per second CPU% = percentage of CPU used during the test Latency = milliseconds Lat% = percent of requests that took longer than X seconds CPU Eff = Rate divided by CPU% - throughput per cpu load Sequential Reads File Blk Num Avg Maximum Lat%

Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s

>10s Eff --- - -

-- -- - 2.4.20-4GB 500 4096 32 10.46 4.810% 22.227 20407.44 0.11800

0.00000 217 Random Reads File Blk Num Avg Maximum Lat%

Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s

>10s Eff --- - -

-- -- - 2.4.20-4GB 500 4096 32 0.75 0.715% 438.919 1926.01 0.00000

0.00000 104 Sequential Writes File Blk Num Avg Maximum Lat%

Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s

>10s Eff --- - -

-- -- - 2.4.20-4GB 500 4096 32 2.94 3.460% 1.798 9288.13 0.02279 0

.00000 85 Random Writes File Blk Num Avg Maximum Lat%

Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s

>10s Eff --- - -

-- -- - 2.4.20-4GB 500 4096 32 0.50 1.081% 0.198 206.05 0.00000 0

.00000 46

After tiobench is finished, run iostat again to look at the disk activity produced by this program. A sample of iostat output is shown here:

[View full width]

# iostat -x /dev/hda1 Linux 2.4.20-4GB avg-cpu: %user %nice %sys %idle 1.73 0.00 0.62 97.65 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz

await svctm %util /dev/hda1 28.38 58.58 2.99 6.19 251.00 518.18 125.50 259.09 83.76 11.50

125.25 14.61 13.42 Linux 2.4.20-4GB (steveb) 12/15/2003 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 0.00 0.00 0.00 40 0 dev3-1 0.15 0.10 2.61 459576 12354928 dev22-2 0.00 0.00 0.00 2852 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 0.00 0.00 0.00 0 0 dev3-1 0.10 1.20 0.00 24 0 dev22-2 0.00 0.00 0.00 0 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 0.00 0.00 0.00 0 0 dev3-1 37.75 0.00 568.80 0 11376 dev22-2 0.00 0.00 0.00 0 0

The statistics that iostat captured after tiobench was run show that the r/s (number of read requests issued to the device per second) went from 0.00 to 2.99, and the w/s (the number of write requests issued to the device per second) went from 0.0 to 6.19. By using other statistics from iostat, such as rkb/s and wkb/s, you can conclude that the test workload is writing more than it is reading.

A useful way to use iostat is to use the interval option, which provides information that might help balance the load among physical hard drives. The d option repeats the display for count times. If no wait interval is specified, the default is 1 second.

The next example shows what happens when the d option is run with an interval set to 20 seconds.

 # iostat d 20 Linux 2.4.20-4GB (steveb)     12/15/2003 Device:    tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn dev3-0    0.00         0.00        0.00         40          0 dev3-1    0.15         0.10        2.61     459576   12354928 dev22-2   0.00         0.00        0.00       2852          0 Device:    tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn dev3-0    0.00         0.00        0.00          0          0 dev3-1    0.10         1.20        0.00         24          0 dev22-2   0.00         0.00        0.00          0          0 Device:    tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn dev3-0    0.00         0.00        0.00          0          0 dev3-1   37.75         0.00      568.80          0      11376 dev22-2   0.00         0.00        0.00          0          0

The output shows three samplings of data collected by iostat. The third sample illustrates that 11,376 blocks were written (Blk_wrtn) during the 20-second interval.

iostat can be used to measure the results of tuning a system by load balancing between disks on a system. The example that follows shows that dev8-1 starts out with 0 blk_wrtn. After the workload has finished, blk_wrtn has increased to 6129208. It also shows that device dev8-0 has a very small amount of activity.

 iostat before workload Linux 2.4.21-1.1931.2.349.2.2.entsmp 02/09/2004 Device:    tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn dev8-0  164.69      1312.58       361.87     150002      41354 dev8-1    2.81         5.71         0.00       652          0 iostat after workload Linux 2.4.21-1.1931.2.349.2.2.entsmp 02/09/2004 Device:    tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn dev8-0   55.12       410.41       146.44    152426      54386 dev8-1  341.88      2828.43     16502.98   1050478     6129208

Now we tune this system to balance the reads and writes from dev8-0 and dev8-1 and then rerun iostat to determine the effectiveness of the balancing. After the system balancing, dev8-0 has blk_read as 1253186 and dev8-1 has blk_read as 1051798. For blk_wrtn, dev8-0 has 6209198 and dev8-1 has 6138576. The workload balancing was very effective in that roughly the same number of reads and writes occurred for each device.

 iostat after the workload balancing Linux 2.4.21-1.1931.2.349.2.2.entsmp 02/09/2004 Device:    tps  Blk_read/s  Blk_wrtn/s  Blk_read  Blk_wrtn dev8-0  121.60      612.59     3035.21   1253186   6209198 dev8-1  110.10      514.15     3000.69   1051798   6138576

For a complete list of iostat options, see the iostat man page.

The following rpm command displays the version of the iostat utility.

 # rpm -qf /usr/bin/iostat sysstat-4.0.17-12

iostat uses /proc/stat to get its information. The following output shows information that is available on a typical system.

 # cat /proc/stat cpu  51736 468 97549 1554177 cpu0 51736 468 97549 1554177 page 118861 2298217 swap 752 1654 intr 2663947 1703930 3850 0 0 0 2920 2 0 2 0 0 16035 18696 0 918510 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 disk_io: (3,0):(902005,1069,9164,900936,8177312) (3,1):(16317,9670,227616,6647,356816) ctxt 5243949 btime 1056965685 processes 602555

The sar utility with the b option reports the I/O and transfer rate. sar -b reports the following I/O rates:

tps. Number of transfers per second issued to device.
rtps. Total number of read requests per second to the device.
wtps. Total number of write requests per second to the device.
bread/s. Total amount of data read from the device in blocks per second.
bwrtn/s. Total amount of data written to the device in blocks per second.

 # sar -b Linux 2.4.20-4GB 02:40:00          tps      rtps      wtps   bread/s   bwrtn/s 02:50:00        89.04      4.91     84.13     82.38   1240.69 03:00:00       125.44      0.03    125.41      0.21   1121.14 03:10:00       118.48      0.03    118.45      0.23   1066.31 03:20:00       114.50      0.03    114.47      0.24   1036.06 03:30:00       135.21      0.03    135.19      0.23   1206.21 03:40:00       135.15      0.03    135.12      0.25   1211.16 03:50:00       127.35      0.03    127.31      0.28   1156.07 04:00:00       145.69      0.04    145.65      0.31   1302.00 04:10:02       152.42      0.21    152.21      1.98   1375.69 04:20:01       148.50      0.60    147.90      6.12   1331.70 04:30:00       113.48      0.36    113.12      3.03   1017.97 04:40:00       108.75      0.50    108.25      4.49    968.44 04:50:00         1.09      0.71      0.38     22.71     24.78 05:00:00         0.51      0.26      0.25      3.11      3.90 05:10:00         0.55      0.18      0.37      2.55     11.66 05:20:00         0.13      0.00      0.13      0.00      2.07 05:30:01         0.60      0.02      0.58      0.61     61.96 05:40:00         2.31      1.84      0.47     39.99      8.30 05:50:00         0.32      0.00      0.32      0.05      5.32 06:00:00         0.44      0.10      0.35      2.94     19.77 06:10:00         0.18      0.04      0.14      0.60      2.72 06:20:00         0.25      0.01      0.24      0.16      3.87 06:30:00         0.10      0.00      0.10      0.00      1.51 06:40:00         0.11      0.00      0.11      0.00      1.64 06:50:00         0.13      0.00      0.13      0.00      2.00 07:00:00         0.09      0.00      0.09      0.00      1.51 07:10:00         0.11      0.00      0.11      0.00      1.68 07:20:00         0.36      0.12      0.23      3.78      4.10 07:30:01         0.38      0.04      0.34      0.77      5.65 07:40:00         0.19      0.00      0.19      0.11      3.12 07:50:00         0.13      0.00      0.13      0.00      2.01 08:00:00         0.10      0.00      0.10      0.00      1.57 08:10:00         0.11      0.00      0.11      0.00      1.68 08:20:00         0.14      0.00      0.14      0.00      2.12 08:30:00         0.09      0.00      0.09      0.00      1.41 08:40:00         0.11      0.00      0.11      0.00      1.71 08:50:00         0.13      0.00      0.13      0.00      2.03 09:00:01         0.09      0.00      0.09      0.00      1.56 09:10:00         0.11      0.00      0.11      0.00      1.68 09:20:00         0.15      0.00      0.15      0.00      2.27 09:30:00         0.09      0.00      0.09      0.00      1.40 09:40:00         0.11      0.00      0.11      0.00      1.72 09:50:00         0.20      0.01      0.19      0.20      3.40 10:00:00       100.04    100.06    100.04    100.06     99.86 10:10:00         4.06      3.27      0.79     92.14     16.17 Average:         1.00      0.28      0.72      5.93      9.92

The sar output shows that the peek of 1375.59 bwrtn/s (the total amount of data written to the device in blocks per second) occurred on the system at 04:10:02.