Project2.Learn Basic Unix

Project 2. Learn Basic Unix

This project will familiarize you with commands used to view and manipulate files and directories. It also covers Unix concepts such as the current working directory, pathnames, mounted volumes, and recursion.

Pathnames

README.txt is a file. There may be any number of files with that name in the file system. To describe one particular README.txt file unambiguously, it is necessary to give the full pathname to that file. A pathname describes the location of a file, starting from the root of the file system, and listing each directory that must be traveled to reach the file. One particular README.txt file can be described unambiguously with the following pathname.

/Users/your-username-here/Documents/importantinfo/README.txt

A pathname that starts at the root of the file system is termed an absolute pathname. However, absolute pathnames require much typing. When one gives directions in real life, especially to local places, it makes sense to assume that the starting point is "right here." Similarly, Unix has a "right here" called the Current Working Directory (CWD). If a pathname does not begin with a slash (/) symbol, the starting point is assumed to be the CWD instead of the file-system root. When you start a new Terminal window, CWD is set to be your home directory (/Users/your-username-here). Thus, the absolute pathname given above is equivalent to the relative pathname.

Documents/importantinfo/README.txt

Absolute and relative pathnames can be used wherever a command requires a filename as an argument. For example, let's display the CWD using the command pwd. (My username is saruman.)

$ pwd /Users/saruman

The / and : Dilemma

Unix uses a forward slash (/) symbol as a separator, so it is not possible to use it in a filename. The Finder uses a colon (:) symbol, and similarly, this cannot be used in a filename. You might see a dilemma here, where one system creates a filename that is illegal in the other. What happens is that the two characters are transposed as the filename is written or viewed in the different systems.

Next, let's list the contents of my Documents directory, using an absolute pathname.

$ ls /Users/saruman/Documents AppleWorks User Data   HouseHold   iChats Connections            Job         importantinfo Letters

This time, we'll use a relative pathname on the assumption that the CWD is /Users/saruman.

$ ls Documents AppleWorks User Data   HouseHold   iChats Connections            Job         importantinfo Letters

Finally, by preceding Documents with a "/" symbol, we are specifying an absolute pathname implying that the directory lies in the file-system root, which it doesn't.

$ ls /Documents ls: /Documents: No such file or directory

Basic Commands

This section gives a brief overview of the most essential Unix commands. To find out more about these and others, use the Unix manual (see Project 3). Most projects in this book are task-focused rather than command-focused and teach how to use a command, or many commands together, to complete a particular task.

Changing the CWD

Sometimes the files you are working on are not in your home directory or are a couple of directories into it. In this situation, it's convenient to set up a new base camp by moving your CWD. Use the command cd to achieve this. Naturally, the argument to cd can be either a relative or an absolute pathname, but must be a directory name and not a filename. First, let's display the CWD and list the directory Sites/images. We use a relative pathname and assume that Sites lies in the current directory.

$ pwd /Users/saruman $ ls Sites/images apache_pb.gif macosxlogo.gif web_share.gif

Next, we change the CWD to Sites/images, enabling us to issue a simple ls command without specifying the directory name.

$ cd Sites/images $ pwd /Users/saruman/Sites/images $ ls apache_pb.gif macosxlogo.gif web_share.gif

View Files with less and cat

Use the command less to view text files on the Terminal. It displays them a pageful at a time. The command less is not an editor; it only displays files.

$ cd ~ $ less Sites/index.html ...

To move forward through the file a page at a time, press the spacebar. To move back and forth one line at a time, use the cursor keys (up arrow and down arrow). To finish viewing and return to the command line, press q. Other useful keystrokes include

• u and d to move up and down a half-page

• Control-g to display a status line

• /word and then Return to forward-search for word

• ?word and then Return to backward-search for word

• n to search for the next occurrence of word

The command cat displays a file all in one go, without pausing at each pageful.

Tip

Issue the command cd with no arguments to change the CWD to your default home directory.

Tip

The tilde (~) symbol is shorthand for the absolute pathname of your home directory. The command ls ~/Art lists the contents of directory Art in your home directory, no matter what the CWD is set to. It's always equivalent to the command ls /Users/[username]/Art.

Learn More

Projects 13 and 14 tell you more about changing your CWD.

Learn More

The projects in Chapter 4 tell you more about editing files in Unix.

Learn More

Projects 21 and 22 tell you more about viewing files.

Copy Files

The command cp makes a copy of a file, taking as arguments the pathname of the file to be copied and the pathname of the new copy, in that order and separated by a space. The copy can be placed in the CWD or any other directory.

Tip

Use option -p with copy to preserve file metadata such as the time stamp, the file owner, and file permissions.

If the file copy is being placed in the same directory as the original file, you must give it a name different from the original's. Here, we create a copy of index.html named index2.html. Let's check what files we have and then apply the copy command.

$ cd ~/Sites $ ls images index.html $ cp index.html index2.html $ ls images index.html index2.html

To copy to another directory, and perhaps rename the copied file too, specify a pathname including a new filename.

$ cp index.html /Users/Shared/home.html

If you're copying to another directory, use that directory's name as your final argument instead of a filename, and the copied file will keep the same name as the original. This procedure makes it easy to copy many files at once to a new directory: Just list all the filenames to be copied as arguments, followed by the name of the directory into which all the files are to be copied.

$ cd images $ ls apache_pb.gif macosxlogo.gif web_share.gif $ cp apache_pb.gif macosxlogo.gif /Users/Shared/ $ ls /Users/Shared/ GarageBand Demo Songs   apache_pb.gif   macosxlogo.gif SC Info                 home.html

Delete Files

Use the command rm to delete (remove) a file. These examples delete the files created by the cp examples above.

$ cd ~/Sites $ ls images index.html index2.html $ rm index2.html $ rm /Users/Shared/home.html $ rm /Users/Shared/apache_pb.gif /Users/Shared/macosxlogo.gif

Tip

With mv, you need not name the destination file if it has the same name as the source file. To move the file index.html to /Users/Shared/index.html, simply type

$ mv index.html ¬    /Users/Shared/

Move Files

Use the command mv to move or rename a file. To move a file, simply give the original pathname and the new pathname as arguments.

First, let's set the CWD to be Sites and see what files it contains.

$ cd ~/Sites $ ls images index.html

Now let's move index.html to the directory /Users/Shared.

$ mv index.html /Users/Shared/index.html

Check that the file has moved.

$ ls images $ ls /Users/Shared/ GarageBand Demo Songs   SC Info        index.html

A file can be renamed as it is moved.

$ mv index.html /Users/Shared/home.html

The command mv is also used to rename a file without moving it, so it is the command we use to rename files.

$ mv index.html home.html

As with copying, many files can be moved to a destination directory in one go.

$ cd images $ mv apache_pb.gif macosxlogo.gif /Users/Shared/

If you follow these examples, don't forget to move the files back to their original locations.

$ mv /Users/Shared/index.html ~/Sites/ $ mv /Users/Shared/apache_pb.gif ~/Sites/images/ $ mv /Users/Shared/macosxlogo.gif ~/Sites/images/

Touch Me

Touch is an easy way to create a new (empty) file, although its real purpose is to alter the timestamp of an existing file.

It's Goodbye, Not Au Revoir

Warning

Unix assumes you know what you are doing. If you remove a file, for example, that file will be removed; it won't be placed in the Trash, and you won't be asked for confirmation. The file will never been seen again. It's goodbye, not au revoir. (This assumes you have the necessary permissions to delete the file.) If you copy file-one to file-two, and file-two already exists, file-two will be overwritten. Gone. The same applies to moving files. Move file-one to file-two, and if file-two already exists, it will be overwritten. The old file-two will be gone. If this behavior makes you a little nervous, option -i may help calm you.

Create two files.

$ touch x y

Try to rename x to y, with and without the option -i. The first line will prompt you, asking whether you really want to overwrite y; the second line won't. (The second line will fail if you previously allowed the first to proceed.)

$ mv -i x y $ mv x y

Using the -i option with the copy command also generates a confirmation prompt.

$ cp -i x y $ cp x y

It works for file removal, too.

$ rm -i x y $ rm x y

In the above examples, option -i specifies interactive mode.

Warning

The Mac OS X file system (called HFS+) allows each file two forks: the data fork, which is the normal data contents of the file, and a special resource fork. The resource fork contains file metadata, such as the creator application, a custom icon, and perhaps a preview if the file holds an image. Unix does not recognize resource forks, so if you were to copy a file using the Unix cp command, only the data fork would be copied: The resource fork is lost.

Having said that, as of Mac OS X 10.4 (Tiger), many Utilities have been written to understand resource forks. To quote the Apple documentation on the subject:

Extended attributes. Tiger introduces a new level of file system sophistication with vfs-level support for extended attributes, inspired by the POSIX.2e proposal. These include *xattr routines to read, write, list, and delete extended attributes. Extended attributes are stored natively on HFS+ as forks and emulated on other file systems via the AppleDouble format (that is, "._filename"). This provides a consistent, Darwin-level API for managing resource forks, metadata, security information, properties, and other attributes.

cp, mv, scp, emacs, vim, pico. Properly handle HFS+ resource forks using the new extended attribute APIs, as do archivers such as tar, rsync, gzip, bzip2, and cpio.

Tip

Use the command echo as an easy way to create a short file containing a line of text. For example,

$ echo "This is file ¬    one" > file1

creates file1 containing the specified text. Project 10 has more examples that use echo.

Multiple Files

Most commands are happy to operate on more than one file at a time, as shown in the examples above. Sometimes it's handy to be able to say "all files" or "all JPEG files" instead of naming them individually. This concept comes under the strange name of globbing or wildcard expansion and is implemented by all shells.

Learn More

Projects 11 and 12 explain the concept of globbing or wildcard expansion.

Commands for Directories

The use of directories to organize and manipulate multiple files is an important Unix skill. Some commands operate on directories the same way they do other files, but there are also some directory-specific commands you must know.

Create a Directory

Use the command mkdir to create a new, empty directory.

$ ls Desktop     Library   Music      Public Documents   Movies    Pictures   Sites $ mkdir Projects

The command ls will show a new directory called Projects.

$ ls Desktop     Library   Music      Projects   Sites Documents   Movies    Pictures   Public

Remove a Directory

Use the command rmdir to delete (or remove) an empty directory.

$ rmdir Projects

The command rmdir won't remove a directory that is not empty. You must use the command rm recursively (see "Recursion" later in this chapter).

Copy and Move a Directory

The commands cp and mv, described earlier in this chapter, can be applied to directories. The command mv can be used for directories in exactly the same way as it is used for files and can move and/or rename a directory. The command cp must be used recursively (see the following section, "Recursion").

Tip

Using mkdir with option -p allows us to create both outer and inner in one command. Without that option, mkdir would complain that outer does not exist, forcing us to create outer first and then outer/inner.

Recursion

Recursion occurs when an operation applied to a directory is reapplied to each item it contains. The recursive use of ls on your home directory, for example, will list the files and directories it contains, the contents of those directories and any directories they contain, and so on until it lists every one of your files. The commands cp and rm have a recursive option where they can be used to copy and remove whole directory hierarchies.

The following example illustrates the use of recursion with ls.

First, we use mkdir to create two directories: outer and, inside it, inner.

$ mkdir -p outer/inner

Now we use the touch command to create files called file1 and file2 in the outer directory and file3 and file4 in the inner directory.

$ touch outer/file1 outer/file2 $ touch outer/inner/file3 outer/inner/file4

Now we list the contents of outer, using ls and activating recursion with option -R.

$ ls -R outer file1   file2   inner outer/inner: file3   file4 $

Notice that ls did not stop once it had listed the contents of outer but continued with inner and listed its contents, too. Many other commands operate recursively, using option -R or -r.

Copy a Directory

To copy the entire contents of a directory, including nested directories and their contents too, use cp in recursive mode by specifying option -R. Here, we copy outer and all of its files, including inner and all of its files, to a new directory called outer-new.

$ cp -R outer outer-new $ ls -R outer-new file1   file2   inner outer-new/inner: file3   file4 $

Remove a Directory

To remove a nonempty directory, use a recursive remove by specifying option -R or -r.

$ rm -ri outer-new examine files in directory outer-new? y remove outer-new/file1? y remove outer-new/file2? y examine files in directory outer-new/inner? y remove outer-new/inner/file3? y remove outer-new/inner/file4? y remove outer-new/inner? y remove outer-new? y $

Remember that rm removes a file completely and does not prompt for confirmation. The recursive use of rm can be very destructive if you accidentally give it the wrong directory to remove. Here, we have used option -i to be on the safe side. If you feel confident, remove outer, too, without the safety net.

$ rm -r outer

Dotty Files

Issue the following commands in your home directory.

$ touch .hidden $ ls

The file you just created is not there! Why? Files that begin with a dot (period) are hidden. A hidden file is no different from a normal file, other than the fact that its name begins with a dot and the fact that ls does not show it. A hidden file is often used to hold configuration information that does not need to be viewed or changed regularly. Hiding prevents visual clutter in the directory listing. View hidden files by applying the -a option to ls.

$ ls -a .                     .bash_history      Movies ..                    .hidden            Music .CFUserTextEncoding   Desktop            Pictures .DS_Store             Documents          Public .Trash                Library            Sites

Now you see .hidden. You will also notice that the first two files listed are dot and dot-dot. There may be other files that start with a dot too, like .trash, which is used by the Finder to hold items you have put in the trash. The files dot and dot-dot are present in every directory and are shorthand (akin to Finder aliases) for the current directory (dot) and its parent directory (dot-dot). When you type dot at the start of a pathname, it's equivalent to typing the absolute pathname of the current directory. When you type dot-dot, it's equivalent to typing the absolute pathname of the directory just above the current directory. Issue the following commands from your home directory (the output is not shown).

$ ls . $ ls ..

Learn More

The dot and dot-dot files are actually hard links to their respective directories. Project 19 tells you more about Unix links.

Learn More

If you wish to know more about permissions, users, and groups, read Projects 7 and 8.

The command ls. lists the contents the current directory and is equivalent to a simple ls. The command ls .. lists the contents of the parent directory, which, from your home directory, should be /Users. Dot and dot-dot can be used anywhere an absolute pathname can. Dot comes in handy when you want to say "here." The three lines below are equivalent.

$ mv Sites/index.html . $ mv Sites/index.html index.html $ mv Sites/index.html ./index.html

Dot-dot is handy when you want to move the CWD up a level . . .

$ cd ..

. . . or two.

$ cd ../..

How to Become the Root User

You are no doubt aware of the concept of file permissions in Mac OS X. You may have tried to edit or delete a file and been denied permission, or have had to install a new application using an administrator account.

With respect to permissions, users can be divided into three groups:

• Normal (nonadministrative) users, who are able to modify files in their home directories but only view or execute files outside them.

• Administrative users, who are also able to create and modify files in certain directories outside their home directory, such as /Applications.

• The all-powerful root user, who can override permissions and who has the power to change everything except your bank balance.

Some projects require write access to system files, so you must become the root user temporarily. You achieve this in one of two ways. To become the root user for the duration of a single command, use the command sudo. Simply type sudo before the command you wish to issue as root. For example, only root can write to the crontab file in /etc.

$ touch /etc/crontab touch: /etc/crontab: Permission denied $

That attempt failed, so let's do the same operation, but running as user root.

$ sudo touch /etc/crontab We trust you have received the usual lecture from the Administrator. It usually boils down to these two things:         #1) Respect the privacy of others.         #2) Think before you type. Password: $

At the password prompt, type your administrator password. Nothing will be echoed back as you type. (You'll not be prompted for the password again if you issue another sudo command within the next five minutes.) The touch command will now (silently) succeed.

A nonadministrator user cannot use sudo. This is by design to prevent normal users from escalating their privileges.

$ sudo touch /etc/crontab Password: loraine is not in the sudoers file. This incident will be reported.

Naughty!

The second way to become the root user is to start a root shell, again using sudo.

$ sudo -s Password: #

Option -s tells sudo to start a new shell running as user root. You'll notice no difference except that you are now root, and your every command will be honored. Unless you have altered the default prompt, it will end with a hash (#) symbol to indicate (warn you of) a root shell. You may now delete essential system files and completely hose Mac OS X. I wouldn't recommend that you do so, but when you're root, nothing is going to argue with you!

Having attained a root shell, do what you need to do, thinking before you type. When you have completed your task, type exit to exit the root shell. Now you can relax and reflect on the fact that in the days before Mac OS X, you effectively did everything as root!

Warning

Running as root gives you full access to Mac OS X system files. Incorrect or ill-advised commands can seriously damage the health of your Mac.

Partitions and Disks

As we've seen, the path separator in Unix, used in pathnames to denote levels in directory hierarchy, is the forward slash (/) symbol. The first level of every absolute pathname, the system disk, has no name; that's why an absolute path always starts with /. In fact, in Unix, no volume (disk or partition) has a name. This is at odds with the Mac OS X Finder, which gives all volumes names. Another difference between Unix and the Finder, and one that can lend confusion, is the way volumes other than the active system disk appear in pathnames. In the Finder, mounted volumes such as CD-ROMs, external drives, and file servers are represented on a peer level with the system hard drive. In Unix, nothing is on a peer level with the system disk; all other volumes are mounted within it, and their pathnames are listed under its root. This leads to an obvious question: "How do I view a mounted disk from the command line?"

When a disk is mounted, it's mounted into a directory, which can be any directory anywhere on the file system but is generally one created specifically for the mount and named appropriately. Darwin automatically mounts all volumes at /Volumes/name-of-vol-in-finder. If you have a mounted volume named Backup, it will appear as /Volumes/Backup. Just treat it like you would any other directory.

$ ls /Volumes Macintosh HD       Backup $ ls /Volumes/Backup Music              System         Users $ cd /Volumes/Backup/Users $ ls Shared             saruman

Further Projects

Refer to these projects to see other essential Unix commands in action:

• grep Project 23

• find Projects 15, 17, 18, and 20

• awk Projects 60 and 62

Refer to these projects to learn more essential Unix:

• Globbing Projects 11 and 12

• Regular expressions Projects 77 and 78

• Hard and soft links Project 19

• Editing files Chapter 4

Refer to these projects to gain practical experience in some commonly performed tasks:

• Finding files Projects 15, 17, 18, and 20

• Searching files by content Project 23

• Compressing files Project 27