Section 12.3. Monitoring System Activity

12.3. Monitoring System Activity

Once you know what is installed on your system, it's always helpful to see how your system is running. The System Profiler report gives access to two logs (console and system) that provide a little bit of information, but two tools, Console and Activity Viewer (both found in /Applications/Utilities), can provide you with quite a bit more information about what's going on in your system, such as how much memory is available, what processes are tying up system resources, and what kind of network activity is going on.

12.3.1. Console

The Console application (shown in Figure 12-3) lets you view all the logfiles on your computer. A logfile is a simple text file that software applications use to report their status. Logfiles are sequential, with new entries appended to the log and timestamped as they occur. Depending on the application, the log might be used solely for error reporting or it might list every mundane task the software performs.

When you launch Console, it gathers the system logs, and any logs from your Home directory, and displays them in the left pane, making it easy to navigate between the logs. If you don't see the left pane, click the Logs icon in the Console's toolbar (or select View Show Log List from the menu bar) to reveal it.

The first two logs you'll see are the system and console logs, but that's just the warm-up. Console also gives you the following areas in which to see the logs:

~/Library/Logs

Contains the logs that are associated with a user. These include the various CrashReporter logs, which give the gory details of any application crashes that you might have experienced. You'll also find the MirrorAgent log detailing the activities of synchronizing your iDisk.

/Library/Logs

Contains logs that are associated with the system at large. For example, you'll find that CrashReporter reports on components that are being run by the system (instead of applications you run) and, if you are unlucky enough to experience a kernel panic, the log for that is found here, too.

Figure 12-3. The Console application displays the various logs available in a sidebar on the left

/var/log

Contains the logs of the various BSD Unix utilities and daemons running on your system. For example, you'll find the logs for the Apache httpd web server and the ipfw firewall here.

With such an extreme amount of data at your fingertips, how can you sort through it all? This is where Apple's use of the Filter box in the menu bar comes in handy. Figure 12-4 shows an example of the user searching for any lines containing "kernel" in the system.log. The quick filtering abilities of the Console utility make it easy for you to look for needles in a haystack. A great use of this feature is to scan your web server's logs to search for requests from a particular IP address.

Another useful tool in the toolbar is the Mark button, which inserts a separator of text, along with the current date and time, into the log view. This allows you to bring up the Console, make a mark in a log, do something with another part of the system, and then come back to see what has happened since you made the mark.

For example, if you're having problems synchronizing your iDisk, you can select the ~/Library/Logs/MirrorAgent.log file in Console, click Mark to stamp it with the current date and time, and then attempt another sync. As MirrorAgent (the process in

Figure 12-4. Filtering for "kernel" in the system log

charge of iDisk synchronization) contacts the .Mac servers and begins synchronizing, you'll see the log entries pop up in Console.

After the synchronization has completed (or failed, in this case), use Mark to stamp the log again. At this point, the information between your two Marks would prove invaluable to the .Mac support team. As you can see, the Mark feature makes it easy to distinguish a particular series of log entries from the rest of the logfile. It's a way for you to make a checkpoint in the logfile before attempting to troubleshoot a problem.

12.3.2. Activity Monitor

To view what's going on with your system right now, nothing beats the Activity Monitor, shown in Figure 12-5. For example, say that Word is taking an awfully long time to open a new file; you could pop open the Activity Monitor to see what the holdup is. Maybe it's just sucking up more memory than what's available? The Activity Monitor gives real-time measurements of resources in use on the system, the processes that are taking up the most memory or CPU cycles, and the system's disk and network activity.

Figure 12-5. The Activity Monitor

The top part of the Activity Monitor window contains a table list of the processes on your system along with the various attributes of those processes. Unlike the Dock or the Force Quit panel ( Force Quit), the Activity Monitor lets you see every process running on your system. The processes in the main window can be filtered either with the text box or the pull-down menu in the menu bar.

The "All Processes, Hierarchically" option provides an interesting view. Using this view, you can see how all the processes on your system are owned by the launchd process, as discussed in Chapter 5. You can also see how GUI applications are owned by the WindowServer process and, if you have a Terminal window open, you can see the various processes owned by it, including the bash shell.

Not only can you monitor every application on the system from the Activity Monitor, but you can also terminate applications. When you select a program and then press the Quit Process button (or use View Quit Process, or Option--Q), a dialog sheet will appear giving you the option of politely quitting the application (this works the same as going to that application and selecting File Quit, or -Q) or using Force Quit.

Figuring Out What a Process Does

It's pretty easy to figure out what many of the processes that show up in Activity Monitor do, particularly with applications like Safari or iTunes, which show up with their icon and name. Others such as slpd, rpc.lockd, or netinfod, however, aren't as easy to figure out simply by looking at them. These are usually the processes that take care of various system functions. If you are curious about what a particular process is, try looking at its manpage. For example, the manpage for netinfod tells you that it is the NetInfo database daemon.

12.3.2.1. Interpreting process information

The impact of a process on your system can be in the amount of memory it consumes, the processor activity, or in the amount of disk I/O that it causes. Some processes can be running without using any processor time and can consume large amounts of memory. Others can consume your processor without any impact on memory usage. The process view gives quite a bit of this information to you all at once. Table 12-1 lists the various columns in the top part of the window and what they mean. The most useful columns to use are %CPU and Real Memory. If you notice your system is sluggish, click the %CPU heading to find out which process is using the most processing power.

Table 12-1. Process attributes reported in the Activity Monitor main window

Attribute	Description
PID	The numerical id of the process
Process Name	The name of the process
User	The user to whom this program belongs
%CPU	The amount of CPU time that the process is taking on your system
# Threads	The number of threads that the process is running
Real Memory	The amount of physical memory used by the process
Virtual Memory	The total amount of memory addressed by the process

To get more detail on a process, select the process in the top part of the window and then hit the Inspect button or select View Inspect Process (-I). This brings up a detailed Inspector window (see Figure 12-6) that gives you more information about the process such as the Mach ports in use by the application, the number of messages being sent via the Mach ports, the files that the application is using, and the various memory statistics, such as faults and page-ins for the process. Many of these statistics are of use only to developers, but they can give you a good idea of the kinds of resources that your applications are using.

Figure 12-6. Inspecting Safari's process

12.3.2.2. Sampling an application

The Process Inspector window gives access to a powerful feature that is of interest mainly to developers who want to take a peek into the inner operation of an application. From the Inspector, hit the Sample button, and another window pops open (shown in Figure 12-7), allowing you to see the call stack of an application, the amount of time spent in each function of an application, and other related information that only a developer could want to see. You can also sample a process from the main Activity Monitor window by selecting View Sample Process (Option--S).

12.3.3. Monitoring Processes from the Command Line

As good as the Console and Activity Monitor are, sometimes the best tool for a job is the Unix command line, particularly when you are logged into a machine remotely

Figure 12-7. Sampling Safari's process using the Activity Monitor's Process Viewer

via ssh (the Secure Shell). Fortunately, there is an abundance of great Unix utilities in Mac OS X, many of which served as the inspiration for their GUI counterparts.

12.3.3.1. Working with logfiles

To work with logfiles on the command line, your primary tools of choice are grep and tail. Named after "get regular expression," grep allows you to quickly search through and return lines in a file that contain a pattern of text. Think of it as the command-line version of the Console's Filter box (discussed earlier), although grep predates the Filter box by a few decades. Example 12-4 shows the use of grep to perform a search for the term "kernel" in the system log (the same search as before, giving you a point of comparison).

By default, grep will search for the exact term you specify, including the letter case. If you're not sure whether the file will contain AirPort, Airport, or AiRpOrT, use grep's -i option to have it ignore case.

Example 12-4. Searching the system log for the pattern "kernel"

 $ grep kernel /var/log/system.log Jun 16 11:15:21 ronin kernel[0]: in_delmulti - ignorning invalid inm (0x656e0000) Jun 16 11:15:24 ronin kernel[0]: in_delmulti - ignorning invalid inm (0x656e00c0) Jun 16 17:03:02 ronin kernel[0]: UniNEnet::monitorLinkStatus - Link is down. Jun 16 17:03:06 ronin kernel[0]: AirPort:  Link DOWN (Client disAssoc 0)

grep gives you quite a bit more power than just the ability to search using a simple term. It actually gives you the ability to use any regular expression. For example, if you wanted to find all the lines that contained the words "kernel" and "AirPort" (in that order), you could use the kernel.*AirPort regex, which, in human, means "look for the term 'kernel,' then some number of any kind of characters followed by the term 'AirPort.'" Example 12-5 shows the result of this regex. You can see that grep outputs only the lines in which both the words "kernel" and "AirPort" appear.

Example 12-5. Using a more detailed regexp with grep

 $ grep kernel.*AirPort /var/log/system.log Jun 19 14:32:53 ronin kernel[0]: AirPort: Link UP: "lifeHertz" - 0011240087dd - Jun 19 14:33:30 ronin kernel[0]: AirPort: Link DOWN (Client disAssoc 0) Jun 19 14:33:37 ronin kernel[0]: AirPort: Link UP: "lifeHertz" - 0011240087dd - Jun 19 14:35:36 localhost kernel[0]: AirPortPCI: Ethernet address 00:0d:93:7f:42:48 Jun 19 14:35:46 ronin kernel[0]: AirPort: Link UP: "lifeHertz" - 0011240087dd -

To monitor what's changing in a logfile, instead of trying to read through the whole logfile from the top, Unix gives you the tail command for looking at just the last part of a file. Example 12-6 shows the use of tail on the system log.

Regular Expressions

Regular expressions (also known as regexes) make up a small, highly specialized language for processing text strings. They can be a bit scary to use at first because of their extremely concise syntax. which makes them look confusing and complicated to the uninitiated. Here are a few of the most commonly used characters in regular expressions:

Character	Description
.	Matches any single character
*	Matches zero or more occurrences of the character immediately preceding it
^	Matches the beginning of a line
$	Matches the end of a line
/	Used to quote the following character, for example, to find a $ character

To learn all about regular expressions, you should pick up Mastering Regular Expressions, Second Edition, by Jeffrey E. F. Friedl (O'Reilly).

Example 12-6. Using tail to view the end of the system log

 $ tail /var/log/system.log Jun 22 08:04:59 ronin configd[33]: posting notification com.apple.system.config.network_change Jun 22 08:04:59 ronin lookupd[1733]: lookupd (version 365) starting - Wed Jun 22 08:04:59 2005 Jun 22 08:05:03 ronin mDNSResponder: NAT Port Mapping (LLQ event port.): timeout Jun 22 08:05:04 ronin mDNSResponder:    -1: DNSServiceResolve("Robert's Music", "_daap._tcp.", "local.") failed: Client id -1 invalid (-65549) Jun 22 08:05:04 ronin mDNSResponder:    -1: DNSServiceResolve("Jason's Music", "_daap._tcp.", "local.") failed: Client id -1 invalid (-65549) Jun 22 08:05:04 ronin mDNSResponder:    -1: DNSServiceResolve("Jason's Music", "_daap._tcp.", "local.") failed: Client id -1 invalid (-65549) Jun 22 08:55:01 ronin sudo:   jldera : TTY=ttyp3 ; PWD=/Users/jldera ; USER=root ;  COMMAND=/opt/local/bin/port install gimp Jun 22 08:58:48 ronin ntpd[96]: time reset 0.220736 s Jun 22 10:46:22 ronin kernel[0]: IOAudioStream[0x1c61900]::clipIfNecessary(  ) - Error: attempting to clip to a position more than one buffer ahead of last clip Jun 22 10:46:22 ronin kernel[0]: IOAudioStream[0x1c61900]::clipIfNecessary(  ) - adjusting clipped position to (8f,2cd9)

By default, tail displays the last 10 lines of the file you apply it to. To change this behavior, you can use the -n option to specify how many lines to go backward into the file. Example 12-7 shows this in action.

Example 12-7. Using the -n argument of tail

 $ tail -n 2 /var/log/install.log Jun 21 12:13:37 ronin Software Update[694]: It took 0.011676 seconds to process the volume "Macintosh HD" Jun 21 12:13:37 ronin Software Update[694]: It took 0.005440 seconds to perform the Installation Check

If you want to see the system log as it changes (for example, if you want to monitor the firewall's log on the command line), you can use the -f option to tail, as shown in Example 12-8.

Example 12-8. Using tail -f on the system log

 $ tail -f -n0 /var/log/system.log Jun 22 15:27:02 ronin ipfw: 12190 Deny TCP 192.168.79.1:2446 192.168.79.60:80 in Jun 22 15:27:05 ronin ipfw: 12190 Deny TCP 192.168.79.1:2446 192.168.79.60:80 in Jun 22 15:27:11 ronin ipfw: 12190 Deny TCP 192.168.79.1:2446 192.168.79.60:80 in

To exit from tail when running with the -f option, hit Control-C.

12.3.3.2. Finding open files

When working with your Mac, you may occasionally encounter an error when trying to open a file that's already in use. In most cases, closing all of your applications or restarting your machine will free up the file, but what if you wanted to track down which program was keeping the file open? Activity Monitor gives you plenty of information about a process's memory and CPU usage, but gives little detail about the files a program is accessing.

Mac OS X's BSD layer includes the lsof tool for just this purpose. This complex command lists open files on the system, as well as the user, process, and full path of each of the open files. lsof's output can be overwhelming at first, but a few command-line switches can help you sort through all of those open files for just the ones you're curious about. To see which files a specific application is using, use lsof's -c switch with this syntax:

     lsof -c processname 

Example 12-9 shows lsof listing the files that are in use by iPhoto.

Example 12-9. Using lsof to view a process's open files

 $ lsof -c iPhoto COMMAND PID   USER   FD     TYPE     DEVICE SIZE/OFF     NODE NAME iPhoto  474 jldera  cwd     VDIR       14,2     1122        2 / iPhoto  474 jldera  txt     VREG       14,2  3746104   472267 /Applications/iPhoto iPhoto  474 jldera  txt     VREG       14,2    17688     3320 /System/Library/Core iPhoto  474 jldera  txt     VREG       14,2    81316     3321 /System/Library/Core iPhoto  474 jldera  txt     VREG       14,2   352454     3319 /System/Library/Core iPhoto  474 jldera  txt     VREG       14,2    19336   250427 /Library/Caches/com. iPhoto  474 jldera  txt     VREG       14,2  5425091     5907 /System/Library/Fram iPhoto  474 jldera  txt     VREG       14,2  2244215     4315 /System/Library/Font iPhoto  474 jldera  txt     VREG       14,2  9826240    11314 /usr/share/icu/icudt iPhoto  474 jldera  txt     VREG       14,2   925696  1237911 /Library/Caches/com. iPhoto  474 jldera  txt     VREG       14,2   991360     4312 /System/Library/Font

Another handy lsof option is -u. When called with the -u option, lsof displays all open files by processes owned by a specified user. For example, to see all files that the user jldera has open, issue the following command:

     $ lsof -u jldera 

You can find much more information about lsof's parameters and capabilities from its manpage. It's a very flexible tool, especially in light of the fact that most Unix input and output operations are handled through a filesystem call. For example, since network sockets are considered a type of file in Mac OS X, they appear open on your system. That means you can use lsof's -i switch to see which network connections are open, much like was shown in Chapter 11 using netstat. Example 12-10's output is quite similar to that shown back in Example 11-7.

Example 12-10. Some output from lsof -i

 $ lsof -i COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME Microsoft   124 jldera   35u  IPv4 0x0340cc6c      0t0  TCP *:3994 (LISTEN) Microsoft   124 jldera   37u  IPv4 0x01c88ad0      0t0  UDP *:rockwell-csp2 iChatAgen   235 jldera    6u  IPv4 0x031cb6c0      0t0  UDP localhost:60971->local iChatAgen   235 jldera    7u  IPv4 0x0264f81c      0t0  TCP 192.168.79.60:52636->2 Mail        268 jldera   25u  IPv4 0x03409ac0      0t0  TCP 192.168.79.60:59775->m SubEthaEd   345 jldera    8u  IPv4 0x0341ac6c      0t0  TCP *:6942 (LISTEN) SubEthaEd   345 jldera   12u  IPv6 0x01c8d820      0t0  TCP *:6942 (LISTEN) OmniOutli   346 jldera   16u  IPv4 0x01efddd0      0t0  UDP *:65080 OmniOutli   346 jldera   17u  IPv4 0x031cbee0      0t0  UDP *:62050 ARDAgent  27536 jldera   16u  IPv4 0x01c88a00      0t0  UDP *:net-assistant AppleVNCS 27537 jldera   23u  IPv4 0x01e1cb08      0t0  TCP *:5900 (LISTEN)

When reading lsof's output for network connections, there are a few items that might need some explanation. The NODE column indicates whether a given connection is using TCP or UDP. The NAME column contains most of the information you'll use to determine what is happening on your system. You'll find two types of entries in the NAME column: entries where the software has already established a connection and entries where the software is waiting for a connection.

When an application is waiting for an incoming network connection, it's said to be listening on a given port. In lsof's output, an app that's listening for a connection will have an entry showing what network address it's listening on, as well as the port it's listening on, separated by a colon. In Example 12-10, the SubEthaEdit application is listening on all network addresses (represented by the asterisk) and on port 6942. To further distinguish the entries that are awaiting connections, lsof adds (LISTEN) to the end of the entry.

You might have noticed that some of the network connections in Example 12-10 have words like net-assistant instead of numbers like 5900. This is because lsof has used the /etc/services file to display the ports by name instead of by number. Those entries that still show a number do not have a corresponding entry for their port in /etc/services. To force lsof to show only the port numbers, use its -P switch. The same can be done for IP addresses being represented by DNS entries if you supply the -n switch.

For those applications that have already established a connection, the entry in lsof's output will be a bit different. Established connections will show the network address and port being used for the connection locally (again, separated by a colon), followed by an arrow pointing to the remote end of the connection. The remote host will also have its address and port shown.