Section 16.15 Monitoring Processes | Real World Linux Security Prentice Hall Ptr Open Source Technology Series

16.15 Monitoring Processes

Almost everyone knows something about using ps to list process status to see what programs are running on the system. The typical invocation would be

 ps -axlww | more

It definitely would be worth the time to read its manual page most carefully. The SIZE field lists the size of the running process in KB. Some of these memory pages may be paged out to "swap space." The RSS field is the Resident Set Size. This is how much of the program currently is residing in memory, in KB.

It is helpful to get to know what these values typically are for your system's daemons and commonly used programs, such as Netscape. You might even store the output of a ps invocation on disk. This may enable you to recognize whether a cracker has installed a Trojan version of one of these due to its size being wrong. Some better techniques for detecting this are discussed in Part IV of this book. I have learned to check the RSS value for Netscape frequently because it seems to have a memory leak that can become serious after a day or so.

Often, it is helpful to filter the output from ps through grep to find processes matching a particular UID, program name, or tty device. The following script has been a favorite of mine for years to do this easily; thanks to David Barker for creating it. It is called pp.

 (/bin/echo \ " F    UID   PID  PPID PRI NI SIZE  RSS WCHAN      STAT TTY   TIME COMMAND";\ /bin/ps -axlww|/bin/grep -i $1|/bin/grep -v -w 'grep'\   | /bin/grep -v -w 'pp') | /bin/more

To have pp provide output for a specified PID, say, 1701, the following command could be issued:

 pp 1701

Besides ps, the top program is quite helpful. It provides a dynamically updating view of the most active processes and often is very helpful in spotting abnormal operation. Its display can be limited to the processes of a single account by issuing the "u" command and then entering the account name.

The fuser program will list the processes using a specified file, file system, or network port. It can be used to determine which processes are doing I/O to a suspicious file. (The pp script, discussed earlier, then could be used to find the parent process of these very suspicious processes.) The following command will list the PIDs of any process that currently has /var/adm/messages open:

 fuser /var/adm/messages

Its output will look something like

 /var/adm/messages:  434

Back in the old days, fuser was useful for seeing which processes had files open on a file system that the SysAdmin wanted to unmount. To list processes with files open on /usr, use the command

 fuser -m /usr

You can use our new pp script to identify this process thusly:

 pp 434

Typical output would be

 F UID PID PPID PRI NI SIZE RSS WCHAN STAT TTY TIME COMMAND 040 0 434 1    0   0  1156 280 do_sel S   ?   0:01 syslogd -m 0

But is this the real syslogd or a cracker's in some other directory? The /proc file system should be thought of as a Swiss Army knife. It can be used for many functions that its creators never thought of.

 tr "\000" " " < /proc/434/cmdline ; echo '' syslogd -m 0

Well, that was not useful. Now, for something really exciting.

 ls -l /proc/434/exe lrwx------ 1 root root 0 May 7 7:49 /proc/434/exe->/sbin/syslogd

You just proved that this process was started by invoking /sbin/syslogd. (This requires a 2.2 or later kernel.) Note that this is the full path name of the executable regardless of how it was entered to invoke it. In other words, it may have been invoked as syslogd and found via $PATH or even invoked via

 cd /sbin      ./syslogd

Are you safe? A cracker could have moved /sbin/syslogd somewhere else, moved his Trojan version there, invoked it, and moved the real one back. This can be checked via

 cmp /proc/434/exe /sbin/syslogd

Lastly, you can invoke md5sum on either or both of these files to see whether they have the same contents that they should. (This assumes that md5sum, the shell, and the kernel have not been compromised.)

The fuser program also can be very helpful to see what processes are using a particular local or remote port or which are connected to a particular remote system. To see all the processes that have connected in via telnet's port 23, issue the following command:

 fuser -n tcp telnet

To see which processes are connected to remote system www.jokesnotwork.com, give the command:

 fuser -n tcp ,www.jokesnotwork.com

To kill anyone's processes that fit this profile, give the following command:

 fuser -k -n tcp ,www.jokesnotwork.com

To see which processes are connected to telnet's port on some other system from your system, the following command will show that:

 fuser -n tcp ,,telnet

16.15.1 Monitoring Load

If the load average, network response time, available bandwidth, or disk space of a system varies significantly from what it normally is, this variance could be an indication of a problem. Crackers tend to use lots of your system's resources trying to gain power. They could be using them to crack passwords, to do a find or ls -lR command to study your system, to download tools to your system to increase penetration (for example, from an ordinary user to root), or to upload data and sources.

They could be compiling their tools or recompiling your kernel after adding a Trojan. (See "Confessions of a Berkeley System Mole" on page 373 for an entertaining discussion on just how easy it is to add Trojans to the kernel.) They could be running daemons. Besides ps and its friends, load average analysis is helpful. Many SysAdmins are familiar with the uptime program that provides this but reading the equivalent file in /proc is more efficient when done repeatedly from a program. The first three fields are the average number of programs ready and wanting to run for the past 1, 5, and 15 minutes. The tload program will do this repeatedly and display the results. The xload program will display a graph of the changing system load via X.

 cat /proc/loadavg

Top