Hands-On with Some Unix Programs
Apple includes hundreds of Unix programs with OS X, from the basic to the baroque. I'll mention a few here, to give you a taste.
Types of Unix Programs
There are several major types of Unix programs. The first type, which I've discussed at length in this chapter, are command-line or "interfaceless" programs. Simple ones accept instructions typed on the command line, and generally either show their results directly in Terminal, or manipulate files based on command arguments. Interactive programs, such as the vi and emacs text editors and the more and less pagers, often run in "full-screen" text mode, responding to keyboard commands until the user finishes with the program and exits. Daemons start up, read configuration settings from a file, and then wait in the background for service requests (frequently across a network). (See the sidebar "Daemons and Logs.")
 |
There are many background programs, called daemons, in Unix (try ps -aux | grep root <RETURN> to see some). Daemons are similar to background applications in OS 9, but are much more common and important in Unix. Many of them record important events during operation using the syslogd program, which saves this information in log files stored in/var/log. (I actually talked about several of these log files in Chapter 14.) The amount of information each program records is often configurable through its configuration file and via the /etc/syslog.conf file (see the manual pages for syslogd and syslog.conf for further info). You can find active logs by typing a command such as ls -lt /var/log | head -5 <RETURN>, which lists the four most recently modified logs. Or, to view the last few events in system.log, type tail /var/log/system.log <RETURN>. (This is actually the same content you would see by viewing the system.log file in Console or a log file utility, as discussed in the previous chapter.)
|
Although Unix gets a lot of mileage out of text interfaces and text manipulation—graphical interfaces are generally not as critical in Unix systems as they are under Mac OS or Windows—there are Unix programs that require (or benefit from) graphical interfaces; they have traditionally used the X11 Window System to provide that graphical display. (I talk about X Window systems below.)
Finally, in addition to traditional text and graphical Unix applications, OS X has introduced a new breed of programs—Aqua configuration tools for Darwin/Unix software. BrickHouse, SharePoints, and MacJanitor, discussed in various places in this book, are examples of third-party configuration utilities that provide a Mac OS X graphical interface to Unix programs. In fact, Apple's System Preferences application is the most familiar example—the Network pane manages Darwin's networking configuration, and the Sharing pane controls the Samba (Windows File Sharing) server, Apache (Personal Web Sharing) server, OpenSSH (Remote Login) server, FTP (FTP Access) server, and CUPS (Printer Sharing) system.
Here are a few example of useful Unix programs; some come with OS X, some are installable using fink, and others can be downloaded manually.
X Window Systems (X11 from Apple or XFree86)
I mentioned earlier that Unix is old; it's been around since long before computers had graphical interfaces. However, nowadays most people find graphical interfaces useful, so most modern Unix systems support the X Window System (also called X11). X11 provides a cross-platform toolkit for applications to display graphics on the local system or on a remote system over a network connection. X11 is primitive in many ways, but it's extremely flexible. In addition, because it's designed on a networking paradigm, X11 programs automatically have the ability to be used on a computer other than the one running them (somewhat like VNC and Timbuktu, described in Chapter 11). XFree86 has long been a popular X11 solution, and Apple provides its own package for OS X, available for free at http://www.apple.com/macosx/x11/.
| Note | The DISPLAY environment variable (type echo $DISPLAY <RETURN> to see your current setting) controls where X programs display their windows. This is normally set to ":0.0", which means the default display on the computer that is actually running the program (localhost). However, if you'd like to run X11 programs and display their windows on another computer, you can do so over an SSH connection. In fact, OpenSSH can set everything up for you—see "X11 and TCP forwarding" in the ssh manual page and check the X11 settings in the ssh configuration files (/etc/ssh_config and /etc/sshd_config under OS X 10.2). Note that your ssh session may stay open after logout if it is still being used for X11 forwarding, as described in the man page. |
Text Editors
Because Unix is a text-based operating system, text editors are critically important for things like editing programs and configuration files, composing mail, and revising documents. Unix systems have three broad families of text editors (much like they have sh-based and csh-based shells). The vi editor and its many derivatives are one branch, the emacs editor and its variants are the second (each with its own faction of supporters, and wars between them), and all others form a third group.
The vi editor is horribly nonintuitive (although great compared to its predecessors, ed and ex, which only displayed one line at a time!); however, it has two virtues: it's extremely powerful (if you know the proper incantations) and it's ubiquitous—vi is available on just about every Unix system. (OS X includes nex/nvi, a rewrite of the ex/vi editor pair.) But if you don't know vi already, it is not worth learning just for use in OS X—use something else instead.
The emacs editor is much larger—the non-graphical version Apple includes in OS X is 4.5MB, plus 50MB of supporting files, versus 300KB for vi—but in fairness emacs is much more than a text editor: it's a full-fledged development environment, with a built-in programming language, mail and news reader, web browser, and an expansive collection of plugins. OS X includes emacs 21.1, and Fink can install the graphical xemacs (via fink install xemacs) if desired. However, again, if you don't know emacs, it's not worth learning just to edit text in OS X.
I say "it's not worth learning" vi or emacs mainly because if you just need to edit text, you can use one of the many graphical text editors available, including OS X's own TextEdit, but most notably the commercial BBEdit (http://www.barebones.com/), which includes a command-line bbedit program to access BBEdit from within Terminal. (If you have BBEdit 7 or later installed, try man bbedit <RETURN> for the command-line tool's manual page). BBEdit includes a variety of Unix integration features, including the ability to run scripts directly, syntax coloring in a variety of languages, and CVS support.
That being said, if you don't have access to Aqua programs like BBEdit, perhaps because you're logged in from another system via ssh, you can do quite a bit with pico, a simple nongraphical text editor included with OS X. To make using pico easier, it constantly shows onscreen menus, which is rare for a command-line program.
Process Management: ps, top, kill, and killall (included in OS X)
Unix is a multi-user system, with background processes running all the time, so Unix-based operating systems like OS X include several commands to see and manage what's going on. The most basic of these is ps (process status). Without options, ps isn't very useful, but ps -aux gives you a snapshot of every program running on the system at the time the command is executed, along with the owner, process ID, start time, and CPU and memory utilization of each.
| Note | Generally, programs writing to files or other programs via stdout (with "<", ">", or "|") ignore the size of your Terminal (shell) window, since they aren't displaying in the window. Unfortunately, ps truncates its output to the screen width even when writing to stdout, so you may get different matches with something like ps -aux | grep BBEdit <RETURN> if you change the width of the Terminal window. To get the full output from ps, use -ww as an argument; ps will not truncate its output. |
To see information similar to what ps provides, but updated every few seconds, use top. By default, top shows summary information on 15 programs, along with overall system status information, including how many processes (programs) are running, system load averages (see the getloadavg manual page for an explanation), and total CPU utilization; it updates its display every second. Ironically, top is so demanding that it often uses over 10 percent of CPU cycles itself, so the -s n argument (which tells top to update the display every n seconds) is often useful, as is the -u argument (which shows the top consumers of CPU cycles). With the -w (wide display) argument, top shows additional columns, and it is smart enough to show more programs if running in a taller window. For example, try typing top -uws 5 <RETURN>.
| Note | You may recognize a strong similarity between the output of top and the information provided by the Process Viewer utility, described in Chapter 7. |
Now that you know what other processes are running, what can you do with this information? The kill command can send a signal to another program (using its Process ID); kill's default signal is -15 (also called SIGTERM or TERM), which tells the receiving program to terminate (like the Quit command in a Mac application). There are many other signals, but unfortunately the handling of signals is poorly standardized. The most important signal is -9 (SIGKILL), a special signal that asks the operating system to force the program to terminate (like using Force Quit in OS X).
To ask a process to shut down, first find its Process ID (PID) from top or ps. Typing kill 100 <RETURN> would ask the process PID 100 to quit (with the default SIGTERM signal). To forcefully end process 100, you would type kill -9 100 <RETURN> or kill -KILL 100 <RETURN>.
| Note | The top program runs continuously until you stop it. In order to type a command (such as a kill command), you need to either open a new Terminal window, or quit top by pressing the Q key. |
| Warning | At the time of this writing, SIGKILL (kill -9) doesn't always work in OS X. This is a serious bug, but it does work most of the time. |
Normally the kill command requires you to provide a Process ID, which you can get from top or ps. However, Apple also provides the killall program, which allows you to kill a process by name. (killall actually searches for a matching process name, finds the PID for that process, and then sends the appropriate kill command for you.) For example, killall Terminal <RETURN> would kill the Terminal application. The -s option shows what killall would do without -s. Try typing killall -s Terminal <RETURN>; you'll see the actual command that would be issued had you not included the -s option. As another example, I currently have three tcsh shells running; typing killall -s tcsh produces the following output:
[g4:~] power% killall -s -9 tcsh kill -KILL 4571 kill -KILL 4569 kill -KILL 4178
| Note | Like most Unix commands, the killall command is case sensitive. So "Terminal" will match Apple's Terminal program, but "terminal" won't. |
Packet Analysis: tcpdump (included in OS X)
When you want to see what your computer is doing on the Internet, you can use a packet sniffer. There are several good commercial ones, but tcpdump is included in OS X and very capable.
For full details, see the tcpdump manual page, but to get a taste, try typing sudo tcpdump -aN -xX -s0 -c6 host www.reppep.com and port 80 <RETURN>. (Provide your admin password when prompted.) The -a option shows hostnames instead of IP numbers when available, and -N uses short names instead of full names. The -x and -X options show sent and received data in hexadecimal (the middle column, which you can ignore) and ASCII (the right column, which is mostly readable) formats. The -s150 option shows the first 150 bytes of each packet (-s0 would show full packets), and -c6 terminates tcpdump after 6 matching packets. To avoid mixing in other traffic, such as e-mail checking or other browser activity, including host www.reppep.com and port 80 tells tcpdump to show only web traffic to and from www.reppep.com.
Listing 15.10: Output from tcpdump when visiting http://www.reppep.com/~pepper/ in a web browser; the right column shows requests to and responses from the server.
|
[g4:~] power% sudo tcpdump -aN -xX -s150 -c6 host www.reppep.com and port 80 tcpdump: listening on en0 15:49:40.822209 g4.49253 > www.http: S 2115499595:2115499595(0) win 32768 <mss1460,nop,wscale 0,nop,nop,timestamp 3257796741 0> (DF) 0x0000 4500 003c 2333 4000 4006 c13f 425c 68c9 E..<#3@.@..?B\h. 0x0010 425c 68c8 c065 0050 7e17 f64b 0000 0000 B\h..e.P~..K.... 0x0020 a002 8000 5678 0000 0204 05b4 0103 0300 ....Vx.......... 0x0030 0101 080a c22e 0885 0000 0000 ............ 15:49:40.822416 www.http > g4.49253: S 2270896276:2270896276(0) ack 2115499596
win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 24052699 3257796741> (DF) 0x0000 4500 003c e516 4000 4006 ff5b 425c 68c8 E..<..@.@..[B\h. 0x0010 425c 68c9 0050 c065 875b 2094 7e17 f64c B\h..P.e.[..~..L 0x0020 a012 e000 67a6 0000 0204 05b4 0103 0300 ....g........... 0x0030 0101 080a 016f 03db c22e 0885 b8a7 fcb1 .....o.......... 15:49:40.822496 g4.49253 > www.http: . ack 1 win 33304 <nop,nop,timestamp
3257796741 24052699> (DF) 0x0000 4500 0034 2334 4000 4006 c146 425c 68c9 E..4#4@.@..FB\h. 0x0010 425c 68c8 c065 0050 7e17 f64c 875b 2095 B\h..e.P~..L.[.. 0x0020 8010 8218 5670 0000 0101 080a c22e 0885 ....Vp.......... 0x0030 016f 03db .o.. 15:49:40.824632 g4.49253 > www.http: P 1:272(271) ack 1 win 33304
<nop,nop,timestamp 3257796741 24052699> (DF) 0x0000 4500 0143 2335 4000 4006 c036 425c 68c9 E..C#5@.@..6B\h. 0x0010 425c 68c8 c065 0050 7e17 f64c 875b 2095 B\h..e.P~..L.[.. 0x0020 8018 8218 577f 0000 0101 080a c22e 0885 ....W........... 0x0030 016f 03db 4745 5420 2f7e 7065 7070 6572 .o..GET./~pepper 0x0040 2f20 4854 5450 2f31 2e30 0d0a 486f 7374 /.HTTP/1.0..Host 0x0050 3a20 7777 772e 7265 7070 6570 2e63 6f6d :.www.reppep.com 0x0060 0d0a 4163 6365 7074 3a20 7465 7874 2f68 ..Accept:.text/h 0x0070 746d 6c2c 2074 6578 742f 706c 6169 6e2c tml,.text/plain, 0x0080 2074 6578 742f 7367 .text/sg 15:49:40.825815 www.http > g4.49253: P 1:310(309) ack 272 win 57920
<nop,nop,timestamp 24052699 3257796741> (DF) 0x0000 4500 0169 e517 4000 4006 fe2d 425c 68c8 E..i..@.@..-B\h. 0x0010 425c 68c9 0050 c065 875b 2095 7e17 f75b B\h..P.e.[..~..[ 0x0020 8018 e240 ab71 0000 0101 080a 016f 03db ...@.q.......o.. 0x0030 c22e 0885 4854 5450 2f31 2e31 2032 3030 ....HTTP/1.1.200 0x0040 204f 4b0d 0a44 6174 653a 204d 6f6e 2c20 .OK..Date:.Mon,. 0x0050 3137 2046 6562 2032 3030 3320 3230 3a34 17.Feb.2003.20:4 0x0060 393a 3431 2047 4d54 0d0a 5365 7276 6572 9:41.GMT..Server 0x0070 3a20 4170 6163 6865 2f32 2e30 2e34 3420 :.Apache/2.0.44. 0x0080 2855 6e69 7829 2050 (Unix).P 15:49:40.825902 www.http > g4.49253: . 310:1758(1448) ack 272 win 57920
<nop,nop,timestamp 24052699 3257796741> (DF) 0x0000 4500 05dc e518 4000 4006 f9b9 425c 68c8 E.....@.@...B\h. 0x0010 425c 68c9 0050 c065 875b 21ca 7e17 f75b B\h..P.e.[!.~..[ 0x0020 8010 e240 a75d 0000 0101 080a 016f 03db ...@.].......o.. 0x0030 c22e 0885 3c21 444f 4354 5950 4520 4854 ....<!DOCTYPE.HT 0x0040 4d4c 2050 5542 4c49 4320 222d 2f2f 5733 ML.PUBLIC."-//W3 0x0050 432f 2f44 5444 2048 544d 4c20 342e 3020 C//DTD.HTML.4.0. 0x0060 5472 616e 7369 7469 6f6e 616c 2f2f 454e Transitional//EN 0x0070 2220 2268 7474 703a 2f2f 7777 772e 7733 "." http://www.w3 0x0080 2e6f 7267 2f54 522f .org/TR/ 15 packets received by filter 0 packets dropped by kernel
|
Network Exploration: nmap (available via Fink)
The nmap program (available via fink) is a powerful tool for network analysis. It offers a host of different probes, can tell what ports (network services) are available on a particular system, and can make an educated guess at what operating system that computer is running based on identifiable characteristics of network responses.
Without arguments, nmap prints a helpful usage message. Some particularly useful options include -v (verbose), -O (attempt to guess remote OS, requires sudo), -F (fast scan, fewer ports), and -P0 (avoid ping tests, for scanning hosts that block ping responses). For example, after installing nmap, try typing sudo nmap -O -F www.nyu.edu <RETURN>; provide your admin password when prompted. This runs nmap against the www.nyu.edu web server.
| Note | Unlike most other command-line programs, nmap isn't flexible about combining multiple options after a single dash; you must list each option separately (e.g., nmap -v -O -F hostname). |
| Warning | Keep in mind that nmap is a powerful tool, used by both crackers and systems administrators to find security holes. Be very careful with it. |
If you like nmap, the netstat -a command lists all open connections and listening ports (which nmap tries to figure out) on the local system (your own computer). (The /etc/services file lists ports that OS X knows about, and what services they are normally used to provide.)
File Transfer: curl (included in OS X)
The curl program can download and upload files using a wide variety of protocols, including HTTP, HTTPS, and FTP. In its simplest usage, curl goes to a URL and downloads the file to standard output; with -o, it downloads to the specified filename, and with -O, it uses the remote filename for the local file. As an example, curl -o index.html http://www.informinit.com/ would download whatever file the web server www.informinit.com home page, and save it as a local file named informinit.html, while curl -O http://www.informinit.com/ would fail, since there's no filename in the URL.
Directory Synchronization: rsync (included in OS X)
The rsync program does intelligent differential mirroring. It scans through master and clone directories, typically over the Internet, comparing the contents of each file in both directories and copying only the changes from the master to the clone. Using rsync is much more efficient for mirroring changed files than a program like curl can be, since it copies only the differences within files, not entire changed files.
| Warning | Since rsync is a Unix program, it doesn't know about resource forks. This means it's great for keeping MP3 collections in sync between home and office, or backing up Word .doc and StuffIt .sit files, but not suitable for Classic applications or other files with resource forks or essential type/creator codes. If you'd like the functionality of rsync with the ability to work with Mac OS resource forks, check out RsyncX, available from http://www.macosxlabs.org/rsyncx/rsyncx.html. |
Web Log Analysis: analog (available via Fink)
Analog is a fast and free analyzer for web server activity logs (including OS X's Apache logs). It's extremely flexible, and can crunch through huge log files in seconds, producing simple HTML charts or plain text pages. A Carbon (GUI) version of Analog is available at http://www.summary.net/soft/analog.html, but Fink can build a pure command-line version as well. If you like Analog, you should also take a look at DNSTran (also available from http://www.summary.net/), which performs hostname lookups for Analog much more quickly than Analog itself, and Report Magic (http://www.reportmagic.org/), which adds more and better chart options, using the perl language built into OS X.
|
|
