Identifying Process Hangs
| Now that we have covered the building blocks of processes and threads, it is time to address process hangs and their potential causes. There is little hope that killing an offending process with -9 (sigkill) will lead to discovering the root cause of a process hang. Neither will rebooting the OS unless you are dealing with a stale file handle. Furthermore, these steps will not prevent these anomalies from reoccurring. However, by applying the knowledge of how processes are created and how resources are used, the root cause can be identified. When a process appears hung, the first step toward a solution is determining certain critical information about the task. Using the ps command, start by determining the task's state, threads, priority, parent, and wait channel. In addition, identify the cumulative CPU time and the initial start time of the task. A holistic approach is needed because although a single ps command is a good start, it will not deliver all the needed information. Let us first determine whether the process is hung because sometimes a process appears to be blocked when it actually is in the middle of a computation or non-blocking I/O. If cumulative CPU time constantly grows, the task's state will most likely be R. In this state, the process is on the run queue and does not have a wait channel. Monitor its cumulative CPU time. If the process remains in the run queue, it might be performing some calculation that takes a while. Even the fastest computers in the world take a while to calculate an infinite loop! Nevertheless, note that a process in the run state could be the normal operation of that program, an application "feature," or a driver problem. If an offending process is consuming system resources at an extraordinary rate and starving production applications, killing the offending process is justified if the process can be killed. However, sometimes a process cannot be killed. When a process has exhausted its timeslice, it is put to sleep() with a given priority. When the priority of the process falls below PZERO, it is in an uninterruptible state and cannot be signaled; however, signals can be queued, and for some operations, this is normal. For others, where the program has hung and never returns, the cause is usually located in the driver or hardware. If the process has a state of D (blocked on I/O), it is uninterruptible and cannot be killed. For example, a process accessing a file over a failed hard NFS mount would be in a state of D while attempting to stat() a file or directory. Uninterruptible processes usually take place when entering I/O calls, at which point the process has called into the kernel, which is in driver code, during which the process cannot receive signals from user space. In this state, a command cannot be signaled even by a SIGKILL (kill -9). It is important to note that signals are queued if not ignored by a sigmask and executed after the code returns from kernel space. Some signals cannot be masked; see the signal man page for more details. Here is an excerpt from the signal man page: ... Using a signal handler function for a signal is called "catching the signal". The signals SIGKILL and SIGSTOP cannot be caught or ignored. ... A zombie process is another process that a user cannot kill. These processes, however, should not be consuming any CPU cycles or memory resources other than the overhead of having the task structure in the kernel's Virtual Address Space (VAS). The main goal of troubleshooting a zombie process is determining why the parent died without reaping its children. In short, you should focus on why and how the parent dies. Listed next are process state codes pulled right out of the source code. ./linux/fs/proc/array.c /* * The task state array is a strange "bitmap" of * reasons to sleep. Thus "running" is zero, and * you can test for combinations of others with * simple bit tests. */ static const char *task_state_array[] = { "R (running)", /* 0 */ "S (sleeping)", /* 1 */ "D (disk sleep)", /* 2 */ "Z (zombie)", /* 4 */ "T (stopped)", /* 8 */ "W (paging)" /* 16 */ }; ...In Scenario 8-1, we demonstrate an instance in which a process cannot be killed. Scenario 8-1: Troubleshooting a Process That Does Not Respond to killA user begins rewinding a tape but realizes that the wrong tape is in the drive. The user tries to kill the job but must wait for the process to finish. Why? The mt command has made an ioctl call to the SCSI tape driver (st) and must wait for the driver to release the process back to user space so that use signals will be handled. # mt -f /dev/st0 rewind # ps -emo state,pid,ppid,pri,size,stime,time,comm,wchan | grep mt D 9225 8916 24 112 20:46 00:00:00 mt wait_for_completion [root@atlorca2 root]# kill -9 9225 [root@atlorca2 root]# echo $? # This produces the return code for the previous command. 0 = success 0 [root@atlorca2 root]# ps -elf | grep 9225 0 D root 9225 8916 0 24 0 - 112 wait_f 20:46 pts/1 00:00:00 mt -f /dev/st0 The mt command has entered a wait channel, and after the code returns from the driver, the signal will be processed. Let's check the pending signals: cat ../9225/status Name: mt State: D (disk sleep) Tgid: 9225 Pid: 9225 PPid: 8916 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 256 Groups: 0 1 2 3 4 6 10 VmSize: 2800 kB VmLck: 0 kB VmRSS: 640 kB VmData: 96 kB VmStk: 16 kB VmExe: 32 kB VmLib: 2560 kB SigPnd: 0000000000000100 <-- SigPnd is a bit mask which indicates the value of the pending signal. Each byte accounts for 4 bits. In this case, the pending signal has a value of 9, so the first bit on the 3rd byte is set. This algorithm is detailed in linux/fs/proc/array.c under the render_sigset_t() function. The following table illustrates this function. Signal : 1 2 3 4 . 5 6 7 8 . 9 10 11 12 . 13 14 15 16 bit value : 1 2 4 8 . 1 2 4 8 . 1 2 4 8 . 1 2 4 8 kill -3 yields bit mask 0000000000000004 kill -9 yields bit mask 0000000000000100 ShdPnd: 0000000000000100 SigBlk: 0000000000000000 SigIgn: 0000000000000000 SigCgt: 0000000000000000 CapInh: 0000000000000000 CapPrm: 00000000fffffeff CapEff: 00000000fffffeff Troubleshooting the hung process involves these steps:
The following scenario demonstrates a way of troubleshooting a process that periodically hangs and then continues. Scenario 8-2: Troubleshooting a Hanging Web BrowserA user complains that her machine is working great except for her Web browsing application. When accessing Web sites, the browser hangs for a few minutes every so often. The user has installed several different versions and tried other browsers to no avail. You ask her the following question: Has it ever worked? The reply is "Yes . . . several days ago it was fine." For the sake of simplicity, we attempt to find the problem with a light Web browser with little overhead. # strace -f -F -r -T -o /tmp/lynx.strace_2 lynx http://www.hp.com Using the vi editor and greping for network calls, such as poll(), we can identify what seems to be a problem right away: :g/poll/p 3660 0.000085 poll([{fd=0, events=POLLIN}], 1, 0) = 0 <0.000020> 3660 0.000186 poll([{fd=0, events=POLLIN}], 1, 0) = 0 <0.000008> 3660 0.000049 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.005154> 3660 0.000043 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.009763> 3660 0.000042 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.008875> 3660 0.000043 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.009264> 3660 0.000042 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.009216> 3660 0.000043 poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, \ 5000) = 1 <0.001146> 3660 0.000081 poll([{fd=0, events=POLLIN}], 1, 0) = 0 <0.000017> 3660 0.000088 poll([{fd=0, events=POLLIN}], 1, 0) = 0 <0.000008> 3660 0.000088 poll([{fd=0, events=POLLIN}], 1, 0) = 0 <0.000022>We see that some poll() calls took over five seconds each. That would explain the Web browser hanging and taking a long time to browse sites. Focusing on the trace, we see the following: 3660 0.000254 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 <0.000044> 3660 0.000095 connect(3, {sa_family=AF_INET, sin_port=htons(53), \ sin_addr=inet_addr("15.50.74.40")}, 28) = 0 <0.000017> 3660 0.000108 send(3, "\245`\1\0\0\1\0\0\0\0\0\0\3www\2hp\3com\0\0\ 34\0\1", 28, 0) = 28 <0.000404> 3660 0.000476 gettimeofday({1097369839, 928119}, NULL) = 0 <0.000005> 3660 0.000049 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 <5.005154> 3660 5.005262 send(3, "\245`\1\0\0\1\0\0\0\0\0\0\3www\2hp\3com\0\0\ 34\0\1", 28, 0) = 28 <0.000426>Checking the man page on poll, we see that it is waiting for an event to take place on a file descriptor. # ls -al /proc/3660/fd/3 -> socket:[52013] This confirms a network issue. After reviewing the /etc/resolv.conf file, we see that 15.50.64.40 is the nameserver. The user contacted her IT department and found that the nameserver configuration for her network had changed. Switching to a different nameserver in the resolv.conf file alleviated the five-second poll() call and resolved the problem. Commands commonly used to troubleshoot a hung process include the following:
In Scenario 8-3, we show a process that appears to be hung but is not. Scenario 8-3: Troubleshooting an Apparent Process HangIn this scenario, a user's goal is to create a file that takes data, automatically compresses it, and sends it to a new file. To perform this task, the user creates a named pipe and issues gzip, redirecting input from the pipe to a new file. The odd part is that the gzip process seems to hang, and the user cannot find the gzip process when searching ps -ef. So you devise an action plan: Re-create the event and trace the process involved.
|
EAN: 2147483647
Pages: 129