Monitoring a Network

As system administrator, you must monitor the use of the machines you administer to make sure that their performance is acceptable and detect issues early on.

To monitor network activity, you can use the tcpdump command, which returns information contained in packets sent and received by network interfaces on your computer, as illustrated in the following figure. In other words, tcpdump lets you monitor the network traffic going in and out of your computer.

To run tcpdump, Mac OS X requires that you run the command as root using sudo. This requirement prevents unauthorized users from running tcpdump on your computer. Remember, packets can contain sensitive information, such as passwords and confidential information contained in email.

To use tcpdump effectively, you should determine what you're looking for to avoid having to sift through tons of irrelevant packets. To that end, tcpdump provides many options and allows you to use Boolean expressions. For example, to print all packets arriving at or departing from mac17.pretendco.com, use the following command:

sudo tcpdump i en0 host mac17

Monitoring network traffic using tcpdump is useful in cases of Denial of Service attacks. For instance, you can use the following command to monitor all incoming traffic destined to port 80 on your computer:

sudo tcpdump i en0 dst port 80

If you detect an unusual number of requests coming from the same source, you can use the firewall rules on your computer to block all traffic coming from that source.

Note

You can create a script that takes output from tcpdump and analyzes it to detect attack patterns and alert the system administrator.

tcpdump is also useful when trying to determine whether packets are sent to the correct IP address and port.

When you run tcpdump, it will keep running until you stop it by pressing Control-C, using the kill command, or closing the Terminal window. If you stop tcpdump and close the Terminal window, you lose the packets that were captured. To store the packets in a file, use the w option when running the command. For example:

sudo tcpdump i en0 w bonjour.dmp dst port 5297 or 5298

After you capture traffic, use the r option to read the packets from the file:

sudo tcpdump r bonjour.dmp

More Info

For more information about tcpdump, refer to its man page.

Using Ethereal

Ethereal is an open-source packet-sniffing tool that runs on an X Window system. This tool has a graphical user interface that lets you monitor packets in a user-friendly manner and offers features such as the ability to trace a set of related packets. In addition, it provides an easy way for creating filters (Boolean expressions) that you can use to display packets of interest.

To use Ethereal, you have to first install X11 for Mac OS X on your computer using the third Mac OS X installation DVD or the X11 package, which you can download from the Apple support website, www.apple.com/support. Once you have installed X11, locate, download, and install Ethereal from www.ethereal.com/download.html.

In addition to displaying packets, Ethereal has powerful analytical tools that display protocol hierarchy and capture options/output statistics, as shown in the following figures.

Monitoring Remotely

There will be times when you'll need to do remote monitoring. When a computer is physically remote (such as in a data center or another building) or simply inaccessible for some reason (they must be network accessible, of course), you will rely upon remote monitoring tools. Two of those tools are ssh, for remote login from the command line, and Apple Remote Desktop (ARD), for remote management through a Mac OS X application.

ARD has three features that are useful for remote monitoring. As shown in the following figure, it allows you to observe and control the remote computer to run monitoring tools, it can pass commands from the administration software to selected machines, and it can check network performance to get a more qualitative report from a remote computer. One issue with ARD is that it currently doesn't provide secure encrypted communications between computers, so it will be most useful within a corporate firewall.

ssh, on the other hand, is a command-line tool that provides secure encrypted login, making it suitable for connections to computers located anywhere. Using ssh, you can run any command-line tool or script on any computer to which you have administration access, including useful tools such as system_profiler, tcpdump, and netstat.

Remote access to a Mac OS X computer using ARD and ssh is disabled by default. To enable remote access using these two methods, you must enable ARD and Remote Login in the Services pane of System Preferences' Sharing pane.

To start ARD via ssh, use the kickstart command. The following example shows how to activate ARD access for the user david and restart the processes:

$ sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/ kickstart -activate -configure -access -on -users david -privs -all -restart -agent -menu

Monitoring With Log Files

Log files are an excellent source of information for troubleshooting and monitoring system and network activity. Log files might contain:

• More detailed information than what you see in error messages

• Informative messages that are not displayed anywhere else

• Progress messages

• Messages useful to a developer or technician assisting you with an issue

Commands and Tools

To monitor log activity, use Console, which provides a powerful search feature and allows you to add markers (log entries containing a timestamp) to separate log entries and make it easier to browse the log. You can also view logs using a text editor, however as shown in the following figure, Console makes it easy to view multiple log files while they are still being written to.

In addition to Console, you can use the tail f command, which displays the contents of a log file. The command keeps running while it waits for additional entries, which the command displays as they are added to the log file. To stop the execution of tail f, press Control-C or close the Terminal window in which the command is running.

When troubleshooting, you can use Console to log the sequence of events as you re-create the issue. The resulting log file can be useful in determining the cause of the issue.

To log a sequence of events:

1.	Open the Console application, located in Applications/Utilities.
2.	Click Logs in the toolbar.
3.	Click the disclosure triangle next to a log to see more available logs.
4.	Select the appropriate log and click Clear.
5.	Perform the actions that produced the issue you are troubleshooting.
6.	Read the messages that appear in the Console window.

Note

If you don't understand the messages recorded in the log, you can still save the log file for a developer or technician helping you troubleshoot the issue.

When monitoring an activity, it's essential that you keep an eye on logs related to the activity. For example, if you've enabled FTP access to your computer and want to keep potential attackers from using it as a network storage device for illegal material, monitor the /var/ log/ftp.log file regularly. You might detect that a host is continuously trying to access your computer at very short intervals (for example, every minute). In such a case, you would use the information in the FTP log to determine the source of the request and block it.

The following is a list of logs that you might want to monitor:

• /Library/Logs/console.log and /var/log/system.log: Use to monitor general system activity.

• /Library/Logs/Software Update.log: Use to determine whether important software updates have been performed.

• /var/log/daily.out: Use to get a daily synopsis of the state of a computer.

• /var/log/ftp.log: Use to monitor FTP activity.

• /var/log/httpd/access_log: Use to monitor personal Web sharing access activity.

• /var/log/httpd/error_log: Use to monitor failed Web sharing access attempts.

• /var/log/install.log: Use to monitor software installation activity.

• /var/log/ipfw.log: Use to monitor firewall activity.

• /var/log/mail.log: Use to monitor email activity.

• /Library/logs/panic.log: Use to monitor kernel panics.

The /etc/syslog.conf has information for the location of different log files.

Automation

One way to automate the process of log monitoring is to schedule commands or scripts that monitor log files to be executed automatically when a certain condition is met.

To automate the process of log monitoring:

1.	Determine what you're looking for in a particular log file. In almost all cases, you'll be looking for string patterns in log entries. For example, you might want to be notified every time the string "failed" appears in a log entry. The built-in command to use for finding patterns is grep. When searching a log file, grep returns the log entries that match the search criteria.
2.	Determine the actions that must be performed when a certain condition is met. For example, you might want an email sent to you every time the string "failed" appears in a log entry. You also might want to be emailed whenever a Denial of Service attack is detected.
3.	Define the commands or scripts to run. For example, if you want to be emailed whenever the string "failed" appears in an entry in system.log, use the following commands: grep i "failed" /var/log/system.log | mail s "failed" [e-mail] Here, the grep command finds the entries or lines in the log file that contain the string "failed login attempt." Then the output from the grep command is redirected as input to the mail command, which you use to send an email via postfix, the built-in program that Mail uses to send and receive email.
4.	Schedule the running of commands or scripts. The following is an example of a useful command combination to monitor the system profile to detect whether hardware components such as memory have been removed: system_profiler SPHardwareDataType | grep i memory Memory: 256 MB This command combination searches the output generated by system_profilerSPHardwareDataType, which returns only core hardware data, for the word memory, and displays the results. Using a data type with system_profiler, such as SPHardwareDataType, is more efficient because system_profiler fetches only the relevant data instead of all system data, which can take a long time. To get a list of all available data types, enter the following: system_profiler listDataTypes

Using Other Third-Party Tools

In addition to the built-in tools mentioned earlier such as tcpdump, Ethereal, tethereal, and Console, you can use a host of third-party tools to monitor network and log activity. These include:

• Little Snitch: This customizable tool notifies you when an application is attempting to connect to the network and/or the Internet. For more information, go to www.obdev.at/products/littlesnitch.

• MacSniffer: This packet-sniffing tool acts as a front end or graphical user interface for tcpdump. MacSniffer displays the information in an easy-to-read form and lets you filter through packets using a library of filters that you can customize. For more information about MacSniffer, go to http://personalpages.tds.net/~brian_hill/macsniffer.html.

Adding Monitoring to Startup Items

System startup items are folders containing scripts or other programs that prepare a Mac OS X computer for normal operation. While launchd has taken over much of the role of starting most services, startup items still have a place in Mac OS X. Some of the system startup items provided by Apple are located in /System/Library/StartupItems. You should not modify the items in this folder, however, you can define your own custom startup items and store them in /Library/StartupItems.

You can add specialized behavior to the booting sequence using startup items. To create a startup item, make a subdirectory named to describe the behavior you are providing, create a shell script or program that has the same name as the subdirectory, and create a file containing a property list and having the name StartupParameters.plist, as illustrated in the following figure.

The StartupParameters.plist file must contain a set of key-value pairs that defines the startup item's provided services and its dependency relationships to other services. The name of the startup item (the folder and executable name) does not necessarily have to be the same as that of a provided service. For example, the script Apache in the folder Apache provides the service HTTP.

When SystemStarter processes a startup item, it looks for an executable file with the name of the containing folder and runs that file with the argument start during system startup. Currently, the start argument is always supplied, because startup items are executed only at system startup. Apple reserves the right to modify SystemStarter to invoke startup items at other times with other arguments. For example, startup items could be executed at system shutdown with an argument of stop.

If you have installed the Developer Tools package, you can use the Property List Editor application in /Developer/Applications/Utilities/ to create an XML-style property list for StartupParameters.plist.

Incorporating Login Hooks

The preferred way to set a login or logout hook is through use of the defaults command with the key LoginHook or LogoutHook and the path of a script as the value. For example:

sudo defaults write com.apple.loginwindow LoginHook /User/david/myscript

Alternatively, you can change the line in the /etc/ttys file that tells the system to launch loginwindow on the Console Terminal.

You can modify that line to add the following options to loginwindow:

Parameter	Description of value
LoginHook	Path of the program to run when a user logs in
LogoutHook	Path of the program to run when a user logs out
HostName	Alternative name to display in the login window (can be empty)
PowerOffDisabled	Disables the Shutdown and Restart buttons; same as Login preferences setting

Here is an example of a loginwindow line from the ttys file that uses the LoginHook option to invoke a program whenever a user logs in:

console "/System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow LoginHook /Users/David/Scripts/mailLoginToAdmin" vt100 on secure onoption="/usr/libexec/getty std.9600"