5.3. Chapter Summary

This chapter presented the various Linux tools that are available to diagnose memory-performance problems. It demonstrated tools that show how much memory an application is consuming ( ps , / proc ) and which functions within the application are allocating that memory ( memprof ). It also covered tools that can monitor the effectiveness of the processor and system cache and memory subsystem ( cachegrind , kcachegrind and oprofile ). Finally, it described a tool that monitors shared memory usage (ipcs). Used together, these tools can track every allocation of memory, the size of these allocations, the functional locations of the allocations in the applications, and how effectively the application is using the memory subsystem when accessing these allocations .

The next chapter moves away from memory to investigate disk I/O bottlenecks.

Chapter 6. Performance Tools: Disk I/O

This chapter covers performance tools that help you gauge disk I/O subsystem usage. These tools can show which disks or partitions are being used, how much I/O each disk is processing, and how long I/O requests issued to these disks are waiting to be processed .

After reading this chapter, you should be able to

Determine the amount of total amount and type (read/write) of disk I/O on a system ( vmstat ).
Determine which devices are servicing most of the disk I/O ( vmstat , iostat , sar ).
Determine how effectively a particular disk is fielding I/O requests ( iostat ).
Determine which processes are using a given set of files ( lsof ).

6.1. Introduction to Disk I/O

Before diving into performance tools, it is necessary to understand how the Linux disk I/O system is structured. Most modern Linux systems have one or more disk drives. If they are IDE drives , they are usually named hda , hdb , hdc , and so on; whereas SCSI drives are usually named sda , sdb , sdc , and so on. A disk is typically split into multiple partitions, where the name of the partition's device is created by adding the partition number to the end of the base device name . For example, the second partition on the first IDE hard drive in the system is usually labeled /dev/hda2 . Each individual partition usually contains either a file system or a swap partition. These partitions are mounted into the Linux root file system, as specified in /etc/fstab . These mounted file systems contain the files that applications read to and write from.

When an application does a read or write, the Linux kernel may have a copy of the file stored into its cache or buffers and returns the requested information without ever accessing the disk. If the Linux kernel does not have a copy of the data stored in memory, however, it adds a request to the disk's I/O queue. If the Linux kernel notices that multiple requests are asking for contiguous locations on the disk, it merges them into a single big request. This merging increases overall disk performance by eliminating the seek time for the second request. When the request has been placed in the disk queue, if the disk is not currently busy, it starts to service the I/O request. If the disk is busy, the request waits in the queue until the drive is available, and then it is serviced.

6.2. Disk I/O Performance Tools

This section examines the various disk I/O performance tools that enable you to investigate how a given application is using the disk I/O subsystem, including how heavily each disk is being used, how well the kernel's disk cache is working, and which files a particular application has " open ."

6.2.1. vmstat (ii)

As you saw in Chapter 2, "Performance Tools: System CPU," vmstat is a great tool to give an overall view of how the system is performing. In addition to CPU and memory statistics, vmstat can provide a system-wide view of I/O performance.

6.2.1.1 Disk I/O Performance-Related Options and Outputs

While using vmstat to retrieve disk I/O statistics from the system, you must invoke it as follows :

vmstat [-D] [-d] [-p partition] [interval [count]]

Table 6-1 describes the other command-line parameters that influence the disk I/O statistics that vmstat will display.

Table 6-1. `vmstat` Command-Line Options

Option	Explanation
`-D`	This displays Linux I/O subsystem total statistics. This option can give you a good idea of how your I/O subsystem is being used, but it won't give statistics on individual disks. The statistics given are the totals since system boot, rather than just those that occurred between this sample and the previous sample.
`-d`	This option displays individual disk statistics at a rate of one sample per `interval` . The statistics are the totals since system boot, rather than just those that occurred between this sample and the previous sample.
`-p partition`	This displays performance statistics about the given partition at a rate of one sample per `interval` . The statistics are the totals since system boot, rather than just those that occurred between this sample and the previous sample.
`interval`	The length of time between samples.
`count`	The total number of samples to take.

If you run vmstat without any parameters other than [interval] and [count] , it shows you the default output. This output contains three columns relevant to disk I/O performance: bo , bi , and wa . These statistics are described in Table 6-2.

Table 6-2. `vmstat` I/O Statistics

Statistic	Explanation
`bo`	This indicates the number of total blocks written to disk in the previous interval. (In `vmstat` , block size for a disk is typically 1,024 bytes.)
`bi`	This shows the number of blocks read from the disk in the previous interval. (In `vmstat` , block size for a disk is typically 1,024 bytes.)
`wa`	This indicates the amount of CPU time spent waiting for I/O to complete. The rate of disk blocks written per second.

When running with the -D mode, vmstat provides statistical information about the system's disk I/O system as a whole. Information about these statistics is provided in Table 6-3. (Note that more information about these statistics is available in the Linux kernel source package, under Documentation/iostats.txt .)

Table 6-3. `vmstat` Disk I/O Statistics

Statistic	Explanation
`disks`	The total number of disks in the system.
`partitions`	The total number of partitions in the system.
`total reads`	The total number of reads that have been requested .
`merged reads`	The total number of times that different reads to adjacent locations on the disk were merged to improve performance.
`read sectors`	The total number of sectors read from disk. (A sector is usually 512 bytes.)
`milli reading`	The amount of time (in ms) spent reading from the disk.
`writes`	The total number of writes that have been requested.
`merged writes`	The total number of times that different writes to adjacent locations on the disk were merged to improve performance.
`written sectors`	The total number of sectors written to disk. (A sector is usually 512 bytes.)
`milli writing`	The amount of time (in ms) spent writing to the disk.
`inprogress IO`	The total number of I/O that are currently in progress. Note that there is a bug in recent versions (v3.2) of `vmstat` in which this is incorrectly divided by 1,000, which almost always yields a 0.
`milli spent IO`	This is the number of milliseconds spent waiting for I/O to complete. Note that there is a bug in recent versions (v3.2) of `vmstat` in which this is the number of seconds spent on I/O rather than milliseconds.

The -d option of vmstat displays I/O statistics of each individual disk. These statistics are similar to those of the -D option and are described in Table 6-4.

Table 6-4. `vmstat` disk I/O Statistics

Statistic	Explanation
`reads: total`	The total number of reads that have been requested.
`reads: merged`	The total number of times that different reads to adjacent locations on the disk were merged to improve performance.
`reads: sectors`	The total number of sectors read from disk.
`reads: ms`	The amount of time (in ms) spent reading from the disk.
`writes: total`	The total number of writes that have been requested for this disk.
`writes: merged`	The total number of times that different writes to adjacent locations on the disk were merged to improve performance.
`writes: sectors`	The total number of sectors written to disk. (A sector is usually 512 bytes.)
`writes: ms`	The amount of time (in ms) spent writing to the disk.
`IO: cur`	The total number of I/O that are currently in progress. Note that there is a bug in recent versions of `vmstat` in which this is incorrectly divided by 1,000, which almost always yields a 0.
`IO: s`	This is the number of seconds spent waiting for I/O to complete.

Finally, when asked to provide partition-specific statistics, vmstat displays those listed in Table 6-5.

Table 6-5. `vmstat` partition I/O Statistics

Statistic	Explanation
`reads`	The total number of reads that have been requested for this partition.
`read sectors`	The total number of sectors read from this partition.
`writes`	The total number of writes that resulted in I/O for this partition.
`requested writes`	The total number of reads that have been requested for this partition.

The default vmstat output provides a coarse indication of system disk I/O, but a good level. The options provided by vmstat enable you to reveal more details about which device is responsible for the I/O. The primary advantage of vmstat over other I/O tools is that it is present on almost every Linux distribution.

6.2.1.2 Example Usage

The number of I/O statistics that vmstat can present to the Linux user has been growing with recent releases of vmstat . The examples shown in this section rely on vmstat version 3.2.0 or greater. In addition, the extended disk statistics provided by vmstat are only available on Linux systems with a kernel version greater than 2.5.70.

In the first example, shown in Listing 6.1, we are just invoking vmstat for three samples with an interval of 1 second. vmstat outputs the system-wide performance overview that we saw in Chapter 2.

Listing 6.1.

[ezolt@wintermute procps-3.2.0]$ ./vmstat 1 3

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in     cs us sy id wa

 1  1      0 197020  81804  29920    0    0   236    25 1017     67  1  1 93  4

 1  1      0 172252 106252  29952    0    0 24448     0 1200    395  1 36  0 63

 0  0      0 231068  50004  27924    0    0 19712    80 1179    345  1 34 15 49

Listing 6.1 shows that during one of the samples, the system read 24,448 disk blocks. As mentioned previously, the block size for a disk is 1,024 bytes, so this means that the system is reading in data at about 23MB per second. We can also see that during this sample, the CPU was spending a significant portion of time waiting for I/O to complete. The CPU waits on I/O 63 percent of the time during the sample in which the disk was reading at ~23MB per second, and it waits on I/O 49 percent for the next sample, in which the disk was reading at ~19MB per second.

Next, in Listing 6.2, we ask vmstat to provide information about the I/O subsystem's performance since system boot.

Listing 6.2.

[ezolt@wintermute procps-3.2.0]$ ./vmstat -D

            3 disks

            5 partitions

53256

total reads

641233

merged reads

4787741

read sectors

343552

milli reading

        14479 writes

        17556 merged writes

       257208 written sectors

      7237771 milli writing

            0 inprogress IO

          342 milli spent IO

In Listing 6.2, vmstat provides I/O statistic totals for all the disk devices in the system. As mentioned previously, when reading and writing to a disk, the Linux kernel tries to merge requests for contiguous regions on the disk for a performance increase; vmstat reports these events as merged reads and merged writes . In this example, a large number of the reads issued to the system were merged before they were issued to the device. Although there were ~640,000 merged reads, only ~53,000 read commands were actually issued to the drives . The output also tells us that a total of 4,787,741 sectors have been read from the disk, and that since system boot, 343,552ms (or 344 seconds) were spent reading from the disk. The same statistics are available for write performance. This view of I/O statistics is a good view of the overall I/O subsystem's performance.

Although the previous example displayed I/O statistics for the entire system, the following example in Listing 6.3 shows the statistics broken down for each individual disk.

Listing 6.3.

[ezolt@wintermute procps-3.2.0]$ ./vmstat -d 1 3

disk ----------reads------------ -----------writes----------- -------IO-------

     total merged sectors     ms  total merged sectors     ms    cur      s

fd0      0      0      0      0      0      0      0      0      0     0

hde

17099

163180

671517

125006   8279   9925 146304 2831237      0    125

hda      0      0      0      0      0      0      0      0      0     0

fd0      0      0      0      0      0      0      0      0      0     0

hde

17288

169008

719645

125918   8279   9925 146304 2831237      0    126

hda      0      0      0      0      0      0      0      0      0     0

fd0      0      0      0      0      0      0      0      0      0     0

hde  17288 169008 719645 125918   8290   9934 146464 2831245      0    126

hda      0      0      0      0      0      0      0      0      0     0

Listing 6.4 shows that 60 (19,059 - 18,999) reads and 94 writes (24,795 - 24,795) have been issued to partition hde3. This view can prove particularly useful if you are trying to determine which partition of a disk is seeing the most usage.

Listing 6.4.

[ezolt@wintermute procps-3.2.0]$ ./vmstat -p hde3 1 3

hde3          reads   read sectors  writes    requested writes

18999

191986

24701

197608

19059

192466

24795

198360

	       19161     193282      24795     198360

Although vmstat provides statistics about individual disks/partitions, it only provides totals rather than the rate of change during the sample. This can make it difficult to eyeball which device's statistics have changed significantly from sample to sample.

6.2.2. iostat

iostat is like vmstat , but it is a tool dedicated to the display of the disk I/O subsystem statistics. iostat provides a per-device and per-partition breakdown of how many blocks are written to and from a particular disk. (Blocks in iostat are usually sized at 512 bytes.) In addition, iostat can provide extensive information about how a disk is being utilized, as well as how long Linux spends waiting to submit requests to the disk.

6.2.2.1 Disk I/O Performance-Related Options and Outputs

iostat is invoked using the following command line:

iostat [-d] [-k] [-x] [device] [interval [count]]

Much like vmstat , iostat can display performance statistics at regular intervals. Different options modify the statistics that iostat displays. These options are described in Table 6-6.

Table 6-6. `iostat` Command-Line Options

Option	Explanation
`-d`	This displays only information about disk I/O rather than the default display, which includes information about CPU usage as well.
`-k`	This shows statistics in kilobytes rather than blocks.
`-x`	This shows extended-performance I/O statistics.
`device`	If a device is specified, `iostat` shows only information about that device.
`interval`	The length of time between samples.
`count`	The total number of samples to take.

The default output of iostat displays the performance statistics described in Table 6-7.

Table 6-7. `iostat` Device Statistics

Statistic	Explanation
`tps`	Transfers per second. This is the number of reads and writes to the drive/partition per second.
`Blk_read/s`	The rate of disk blocks read per second.
`Blk_wrtn/s`	The rate of disk blocks written per second.
`Blk_read`	The total number of blocks read during the interval.
`Blk_wrtn`	The total number of blocks written during the interval.

{% if main.adsdop %}{% include 'adsenceinline.tpl' %}{% endif %}

When you invoke iostat with the -x parameter, it displays extended statistics about the disk I/O subsystem. These extended statistics are described in Table 6-8.

Table 6-8. `iostat` Extended Disk Statistics

Statistic	Explanation
`rrqm/s`	The number of reads merged before they were issued to the disk.
`wrqm/s`	The number of writes merged before they were issued to the disk.
`r/s`	The number of reads issued to the disk per second.
`w/s`	The number of writes issued to the disk per second.
`rsec/s`	Disk sectors read per second.
`wsec/s`	Disk sectors written per second.
`rkB/s`	Kilobytes read from disk per second.
`wkB/s`	Kilobytes written to disk per second.
`avgrq-sz`	The average size (in sectors) of disk requests.
`avgqu-sz`	The average size of the disk request queue.
`await`	The average time (in ms) for a request to be completely serviced. This average includes the time that the request was waiting in the disk's queue plus the amount of time it was serviced by the disk.
`svctm`	The average service time (in ms) for requests submitted to the disk. This indicates how long on average the disk took to complete a request. Unlike `await` , it does not include the amount of time spent waiting in the queue.

iostat is a helpful utility, providing the most complete view of disk I/O performance statistics of any that I have found so far. Although vmstat is present everywhere and provides some basic statistics, iostat is far more complete. If it is available and installed on your system, iostat should be the first tool to turn to when a system has a disk I/O performance problem.

6.2.2.2 Example Usage

Listing 6.5 shows an example iostat run while a disk benchmark is writing a test file to the file system on the /dev/hda2 partition. The first sample iostat displays is the total system average since system boot time. The second sample (and any that would follow) is the statistics from each 1-second interval.

Listing 6.5.

[ezolt@localhost sysstat-5.0.2]$ ./iostat -d 1 2

Linux 2.4.22-1.2188.nptl (localhost.localdomain)        05/01/2004



Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

hda               7.18       121.12       343.87    1344206    3816510

hda1              0.00         0.03         0.00        316         46

hda2              7.09       119.75       337.59    1329018    3746776

hda3              0.09         1.33         6.28      14776      69688

hdb               0.00         0.00         0.00         16          0



Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

hda             105.05         5.78     12372.56         16      34272

hda1              0.00         0.00         0.00          0          0

hda2            100.36         5.78     11792.06         16      32664

hda3              4.69         0.00       580.51          0       1608

hdb               0.00         0.00         0.00          0          0

One interesting note in the preceding example is that /dev/hda3 had a small amount of activity. In the system being tested , /dev/hda3 is a swap partition. Any activity recorded from this partition is caused by the kernel swapping memory to disk. In this way, iostat provides an indirect method to determine how much disk I/O in the system is the result of swapping.

Listing 6.6 shows the extended output of iostat .

Listing 6.6.

[ezolt@localhost sysstat-5.0.2]$ ./iostat -x -dk 1 5 /dev/hda2

Linux 2.4.22-1.2188.nptl (localhost.localdomain)        05/01/2004

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s     wkB/s

avgrq-sz avgqu-sz   await  svctm  %util

hda2        11.22  44.40  3.15  4.20  115.00  388.97    57.50    194.49

68.52     1.75  237.17  11.47   8.43



Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s     wkB/s

avgrq-sz avgqu-sz   await  svctm  %util

hda2         0.00 1548.00  0.00 100.00    0.00 13240.00     0.00   6620.00

132.40    55.13  538.60  10.00 100.00



Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s     wkB/s

avgrq-sz avgqu-sz   await  svctm  %util

hda2         0.00 1365.00  0.00 131.00    0.00 11672.00     0.00   5836.00

89.10    53.86  422.44   7.63 100.00



Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s   rkB/s      wkB/s

avgrq-sz avgqu-sz   await  svctm  %util

hda2         0.00 1483.00  0.00 84.00    0.00 12688.00    0.00    6344.00

151.0     39.69  399.52  11.90 100.00



Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s     wkB/s

avgrq-sz avgqu-sz   await  svctm  %util

hda2         0.00 2067.00  0.00 123.00    0.00 17664.00     0.00   8832.00

143.61    58.59  508.54   8.13 100.00

In Listing 6.6, you can see that the average queue size is pretty high (~237 to 538) and, as a result, the amount of time that a request must wait (~422.44ms to 538.60ms) is much greater than the amount of time it takes to service the request (7.63ms to 11.90ms). These high average service times, along with the fact that the utilization is 100 percent, show that the disk is completely saturated .

The extended iostat output provides so many statistics that it only fits on a single line in a very wide terminal. However, this information is nearly all that you need to identify a particular disk as a bottleneck.

6.2.3. sar

As discussed in Chapter 2, "Performance Tools: System CPU," sar can collect the performance statistics of many different areas of the Linux system. In addition to CPU and memory statistics, it can collect information about the disk I/O subsystem.

6.2.3.1 Disk I/O Performance-Related Options and Outputs

When using sar to monitor disk I/O statistics, you can invoke it with the following command line:

sar -d [ interval [ count ] ]

Typically, sar displays information about the CPU usage in a system; to display disk usage statistics instead, you must use the -d option. sar can only display disk I/O statistics with a kernel version higher than 2.5.70. The statistics that it displays are described in Table 6-9.

Table 6-9. `sar` Device Statistics

Statistic	Explanation
`tps`	Transfers per second. This is the number of reads and writes to the drive/partition per second.
`rd_sec/s`	Number of disk sectors read per second.
`wr_sec/s`	Number of disk sectors written per second.

The number of sectors is taken directly from the kernel, and although it is possible for it to vary, the size is usually 512 bytes.

6.2.3.2 Example Usage

In Listing 6.7, sar is used to collect information about the I/O of the devices on the system. sar lists the devices by their major and minor number rather than their names .

Listing 6.7.

[ezolt@wintermute sysstat-5.0.2]$ sar -d 1 3

Linux 2.6.5 (wintermute.phil.org)       05/02/04



16:38:28          DEV       tps  rd_sec/s  wr_sec/s

16:38:29       dev2-0      0.00      0.00      0.00

16:38:29      dev33-0    115.15    808.08   2787.88

16:38:29     dev33-64      0.00      0.00      0.00

16:38:29       dev3-0      0.00      0.00      0.00



16:38:29          DEV       tps  rd_sec/s  wr_sec/s

16:38:30       dev2-0      0.00      0.00      0.00

16:38:30      dev33-0    237.00   1792.00      8.00

16:38:30     dev33-64      0.00      0.00      0.00

16:38:30       dev3-0      0.00      0.00      0.00



16:38:30          DEV       tps  rd_sec/s  wr_sec/s

16:38:31       dev2-0      0.00      0.00      0.00

16:38:31      dev33-0    201.00   1608.00      0.00

16:38:31     dev33-64      0.00      0.00      0.00

16:38:31       dev3-0      0.00      0.00      0.00



Average:          DEV       tps  rd_sec/s   wr_sec/s

Average:       dev2-0      0.00      0.00       0.00

Average:      dev33-0    184.62   1404.68     925.75

Average:     dev33-64      0.00      0.00       0.00

Average:       dev3-0      0.00      0.00       0.00

sar has a limited number of disk I/O statistics when compared to iostat . However, the capability of sar to simultaneously record many different types of statistics may make up for these shortcomings.

6.2.4. lsof (List Open Files)

lsof provides a way to determine which processes have a particular file open. In addition to tracking down the user of a single file, lsof can display the processes using the files in a particular directory. It can also recursively search through an entire directory tree and list the processes using files in that directory tree. lsof can prove helpful when narrowing down which applications are generating I/O.

6.2.4.1 Disk I/O Performance-Related Options and Outputs

You can invoke lsof with the following command line to investigate which files processes have open:

lsof [-r delay] [+D directory] [+d directory] [file]

Typically, lsof displays which processes are using a given file. However, by using the +d and +D options, it is possible for lsof to display this information for more than one file. Table 6-10 describes the command-line options of lsof that prove helpful when tracking down an I/O performance problem.

Table 6-10. `lsof` Command-Line Options

Option	Explanation
`-r delay`	This causes `lsof` to output statistics every `delay` seconds.
`+D directory`	This causes `lsof` to recursively search all the files in the given directory and report on which processes are using them.
`+d directory`	This causes `lsof` to report on which processes are using the files in the given directory.

lsof displays the statistics described in Table 6-11 when showing which processes are using the specified files.

Table 6-11. `lsof` File Statistics

Statistic	Explanation
`COMMAND`	The name of the command that has the file open.
`PID`	The PID of the command that has the file open.
`USER`	The user who has the file open.
`FD`	The file descriptor of the file, or `tex` for a executable, `mem` for a memory mapped file.
`TYPE`	The type of file. `REG` for a regular file.
`DEVICE`	Device number in major, minor number.
`SIZE`	The size of the file.
`NODE`	The inode of the file.

Although lsof does not show the amount and type of file access that a particular process is doing, it at least displays which processes are using a particular file.

6.2.4.2 Example Usage

Listing 6.8 shows lsof being run on the /usr/bin directory. This run shows which processes are accessing all of the files in /usr/bin .

Listing 6.8.

[ezolt@localhost manuscript]$  /usr/sbin/lsof -r 2 +D /usr/bin/

COMMAND    PID  USER  FD   TYPE DEVICE   SIZE   NODE NAME

gnome-ses 2162 ezolt txt    REG    3,2 113800 597490 /usr/bin/gnome-session

ssh-agent 2175 ezolt txt    REG    3,2  61372 596783 /usr/bin/ssh-agent

gnome-key 2182 ezolt txt    REG    3,2  77664 602727 /usr/bin/gnome-keyring-daemon

metacity  2186 ezolt txt    REG    3,2 486520 597321 /usr/bin/metacity

gnome-pan 2272 ezolt txt    REG    3,2 503100 602174 /usr/bin/gnome-panel

nautilus  2280 ezolt txt    REG    3,2 677812 598239 /usr/bin/nautilus

magicdev  2287 ezolt txt    REG    3,2  27008 598375 /usr/bin/magicdev

eggcups   2292 ezolt txt    REG    3,2  32108 599596 /usr/bin/eggcups

pam-panel 2305 ezolt txt    REG    3,2  45672 600140 /usr/bin/pam-panel-icon

gnome-ter 3807 ezolt txt    REG    3,2 289116 596834 /usr/bin/gnome-terminal

less      6452 ezolt txt    REG    3,2 104604 596239 /usr/bin/less

=======

COMMAND    PID  USER  FD   TYPE DEVICE   SIZE   NODE NAME

gnome-ses 2162 ezolt txt    REG    3,2 113800 597490 /usr/bin/gnome-session

ssh-agent 2175 ezolt txt    REG    3,2  61372 596783 /usr/bin/ssh-agent

gnome-key 2182 ezolt txt    REG    3,2  77664 602727 /usr/bin/gnome-keyring-daemon

metacity  2186 ezolt txt    REG    3,2 486520 597321 /usr/bin/metacity

gnome-pan 2272 ezolt txt    REG    3,2 503100 602174 /usr/bin/gnome-panel

nautilus  2280 ezolt txt    REG    3,2 677812 598239 /usr/bin/nautilus

magicdev  2287 ezolt txt    REG    3,2  27008 598375 /usr/bin/magicdev

eggcups   2292 ezolt txt    REG    3,2  32108 599596 /usr/bin/eggcups

pam-panel 2305 ezolt txt    REG    3,2  45672 600140 /usr/bin/pam-panel-icon

gnome-ter 3807 ezolt txt    REG    3,2 289116 596834 /usr/bin/gnome-terminal

less      6452 ezolt txt    REG    3,2 104604 596239 /usr/bin/less

In particular, we can see that process 3807 is using the file /usr/bin/gnome-terminal . This file is an executable, as indicated by the txt in the FD column, and the name of the command that is using it is gnome-terminal . This makes sense; the process that is running gnome-terminal must therefore have the executable open. One interesting thing to note is that this file is on the device 3,2, which corresponds to /dev/hda2 . (You can figure out the device number for all the system devices by executing ls -la /dev and looking at the output field that normally displays size.) Knowing on which device a file is located can help if you know that a particular device is the source of an I/O bottleneck. lsof provides the unique ability to trace an open file descriptor back to individual processes; although it does not show which processes are using a significant amount of I/O, it does provide a starting point.

Flylib.com

Category list

Similar pages

5.3. Chapter Summary