4.8 Tracking Individual Processes

You saw how to use ps in Section 1.16 for listing process characteristics at any given time. ps is good for getting a snapshot of the current processes, but it does little to tell you how processes change over time. Using ps alone, it's not terribly easy to pick out processes that are using too much CPU time or memory.

The top program displays the current system status and many of the fields that you might see in a ps listing, but it also updates the display every second. By default, top shows the most active processes (that is, those currently taking up the most CPU time).

You can send commands to top with keystrokes. These are some of the most important commands:

SPACEBAR Updates the display immediately
M Sorts by current resident memory usage
T Sorts by total (cumulative) CPU usage
P Sorts by current CPU usage (the default)
u Displays only one user 's processes
? Displays a usage summary for all top commands

If you want to know how much CPU time a process uses during its entire lifetime, use time . (If your shell has a built-in time command, you may need to run /usr/bin/time to get the full output shown below.) For example, to measure the CPU time used by ls , run this command:

 time ls

After ls terminates, you should get some output like this:

 0.05user 0.09system 0:00.44elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (125major+51minor)pagefaults 0swaps

User time is the number of seconds that the CPU has spent running the program's own code. On modern processors, some commands run so quickly that you may get a zero value here because the actual CPU time is so low that time rounds down to zero.
System time is how long the kernel spends doing the process's work (for example, reading files and directories).
Elapsed time is the total time it took to run the process from start to finish, including the time that the CPU spent doing other tasks . This number is not very useful for performance measurements.

See Section 4.10 for a description of the other fields.

4.8.1 Finding Open Files with lsof

The lsof command lists open files and the processes that are using these open files. Because Unix places such a great emphasis on files, lsof is among the most useful tools for finding trouble spots. lsof does not stop at regular files ” it can list network sockets, dynamic libraries, pipes, and more.

Running lsof on the command line usually produces a tremendous amount of output. Here is a fragment of what you might see:

 COMMAND     PID   USER   FD   TYPE     DEVICE     SIZE     NODE NAME init          1   root  cwd    DIR        3,1     4096        2 / init          1   root  rtd    DIR        3,1     4096        2 / init          1   root  txt    REG        3,1    24480    16212 /sbin/init init          1   root  mem    REG        3,1   999542    32329 /lib/ld-2.3.2.so init          1   root  mem    REG        3,1  1251176    32208 /lib/libc-2.3.2.so init          1   root   10u  FIFO        3,1             97634 /dev/initctl ... ... vi         3511  juser  cwd    DIR        3,4     4096   163866 /home/juser/w vi         3511  juser  rtd    DIR        3,1     4096        2 / vi         3511  juser  txt    REG        3,3   327048   561520 /usr/bin/vi ... vi         3511  juser    3rW  REG        3,4    18993   163867 /home/juser/w/c

The fields in this output are as follows :

COMMAND The command name for the process that holds the file descriptor.
PID The process ID.
USER The user running the process.
FD The file descriptor or the purpose of the open file. A file descriptor is a number that a process uses in conjunction with the system libraries to identify and manipulate a file.
TYPE The file type (regular file, directory, socket, etc.).
DEVICE The major and minor number of the device that holds the file.
SIZE The file's size.
NODE The file's inode number.
NAME The filename.

The lsof(1) manual page contains a full list of what you might see for each field, but you should be able to guess just by looking at the output. For example, look at the entries with cwd in the FD field. These lines indicate current working directories of the processes. Another example is the very last line, showing a file that the user is currently editing with vi .

There are two basic approaches to running lsof :

List everything and pipe the output to a command like less , and then run a search to find what you're looking for.
Narrow down the list that lsof provides with command-line options.

If you use command-line options, you can provide a filename as an argument, and lsof will list only the entries that match the argument. For example, the following command displays entries for open files in /usr :

 lsof /usr

To list the open files for a particular process ID, run this command:

 lsof -p  pid

lsof has dozens of other options; lsof -h provides a short summary. Most options pertain to the output format. Make sure that you look at Section 6.5.1 for lsof network features.

Note	lsof is highly dependent on kernel information. If you upgrade your kernel, you may need to upgrade lsof .

4.8.2 Tracing Program Execution with strace and ltrace

With the exception of time , the tools you have seen so far examine active processes. However, if you have no idea why a program dies almost immediately after starting up, even lsof won't help you. You would have a difficult time even running lsof concurrently with a failed command.

The strace (system call trace) and ltrace (library trace) commands can help you discover what a program attempts to do. These tools produce extraordinarily large amounts of output, but once you know what to look for, you will be in a good position for tracking down problems.

A system call is a privileged operation that the kernel must perform for a process, such as opening and reading data from a file. strace prints all of the system calls that a process makes. To get the idea, run this command:

 strace cat /dev/null

The first part of the output deals primarily with loading shared libraries. You can ignore this stuff:

 execve("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0)                                  = 0x804b348 open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY)      = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3)                                = 0 open("/lib/libc.so.6", O_RDONLY)        = 3 read(3, "7ELF  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
  execve ("/bin/cat", ["cat", "/dev/null"], [/* 52 vars */]) = 0 brk(0) = 0x804b348 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG0644, st_size=14834, ...}) = 0 old_mmap(NULL, 14834, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 1024) = 1024 
 0^"..., 1024) = 1024

In addition, you can skip by all of the mmap output until you get to the lines that look like this:

 fstat64(1, {st_mode=S_IFCHR0666, st_rdev=makedev(1, 3), ...}) = 0 open("/dev/null", O_RDONLYO_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFCHR0666, st_rdev=makedev(1, 3), ...}) = 0 brk(0)                                  = 0x804b348 brk(0x804c348)                          = 0x804c348 brk(0)                                  = 0x804c348 brk(0x804d000)                          = 0x804d000 read(3, "", 4096)                       = 0 close(3)                                = 0 close(1)                                = 0 _exit(0)                                = ?

Now you see some interesting things. First, look at the open() call, which opens a file. The 3 as a result means success ( 3 is a file descriptor). Then, near the end, you see where cat reads from /dev/null (the read() call ” notice that the file descriptor is 3 ). Then there is nothing more to read, so the program closes the file descriptor and exits.

So what happens when there's a problem? Try strace cat not_a_file instead and examine the open() call:

 open("not_a_file", O_RDONLYO_LARGEFILE) = -1 ENOENT (No such file or directory)

Here, strace not only reports the error at the system-call level, but it also gives you a small description of the error.

Missing files are the most common problems with Unix programs, so if the syslog and other log information isn't very helpful, and you have nowhere else to turn , strace can be of great use. You can even use strace on daemons that detach themselves . Here's an example:

 strace -o crummyd_strace -ff crummyd

In this example, the -o option to strace logs the action of any child process that crummyd spawns into crummyd_strace. pid , where pid is the process ID of the child process.

The ltrace command tracks shared library calls. The output is similar to strace , but be warned , there are usually many more shared library calls than system calls. See Section 8.1.4 for more information on shared libraries ( ltrace doesn't work on statically linked binaries).