Examining File Contents

Managing Files and Directories

Having mastered navigating about the filesystem, it's now time to learn how to interact with some of the files. We'll start with basic file manipulation commands that do things such as rename files and make copies of them, and then cover more complicated things such as finding files, extracting portions of their contents, and archiving them.

As mentioned previously, Unix commands tend to be small, single-function commands that can be combined to form more complex functionality. You see this demonstrated by many commands in this chapter; their functionality might seem oddly limited if you're not used to the Unix command philosophy. By the time you're finished with this chapter, however, you should begin to see how the commands could be fit together, and you should be able to abstract what you learn here to most Unix commands.

Moving and Renaming Files: mv

If you're a longtime Macintosh user, you are probably familiar with the notion of moving files about by way of the Macintosh's drag-and-drop formalism. Unix and its command line might not seem like a particularly appealing way to deal with moving files having to type the names and paths to directories can't possibly be much fun! There's no denying that there are certain tasks for which the drag-and-drop way works much better than the command line. But, although you might not have thought about it, there are also situations in which drag-and-drop makes your life much more difficult. Interestingly, these are frequently situations in which the command line works particularly well; for example, when a folder contains many files of the same type, and you're interested in using a number of them that are related by name rather than by icon position. In a situation such as this, rearranging things in the Finder, or Shift-clicking your way through the list of files to pick the ones you want, is usually less efficient than choosing them from the command line by using a shell filename wildcard. Similarly, it's frequently faster to type a filename, if you know it, than to scroll around in a Finder window looking for the file. For these reasons, as well as conveniences that really become apparent only from experience rather than explanation, the command line makes for a useful complement to the Finder for certain operations.

Renaming files in Unix is accomplished with the mv (move) command. It might seem odd at first that renaming a file is accomplished by moving it, but it makes sense in the Unix philosophy of accomplishing things in simple, abstract ways. Why create two commands that do essentially the same thing, when one command can do both with the same syntax, the same way? To rename a file from one name to another, simply use mv <oldfilename> <newfilename>. For example, if you're in a directory with a file named mynewfile, and you want to rename it as myoldfile, you might do something like this:

 brezup:nermal Documents $ ls lynx       lynx.cfg   mynewfile  test brezup:nermal Documents $ mv mynewfile myoldfile brezup:nermal Documents $ ls lynx       lynx.cfg   myoldfile  test 

Remember that most commands can take absolute paths or relative paths to files? Well, being in the same directory and using just the filenames is using the relative paths. On the other hand, you can accomplish the same thing using the absolute paths to the files. Starting from where we left off in the previous example, this might look like the following:

 brezup:nermal Documents $ pwd /Users/nermal/Documents brezup:nermal Documents $ mv /Users/nermal/Documents/myoldfile/Users/nermal/Documents /myevenolderfile brezup:nermal Documents $ ls lynx       lynx.cfg   myevenolderfile  test 

Because you can do that, why should you need to be in the same directory as the files at all? As a matter of fact, because you can specify the full paths to the files, what is to stop you from changing something other than the filename when you use the mv command? What if you decide to change one of the directories in the full path while you're at it? For example, let's look at what happens if you move a file from the Documents directory at the same time that you change its name:

 brezup:nermal Documents $ cd ~ brezup:nermal nermal $ ls Desktop                   Network Trash Folder      chown-output Documents                 Pictures                  myfile Library                   Public                    output-sample6 Movies                    Sites                     su-output Music                     TheVolumeSettingsFolder   typescripts brezup:nermal nermal $ ls Documents lynx       lynx.cfg   myevenolderfile  test brezup:nermal nermal $ mv /Users/nermal/Documents/myevenolderfile/Users/nermal/Public /myolderfile brezup:nermal nermal $ ls Documents/ lynx     lynx.cfg test brezup:nermal nermal $ ls Public/ Drop Box   myolderfile 

The end result is that Unix's abstraction of file access and naming causes the full path to the file to be, essentially, the full proper name of the file. "Renaming" it using the mv command can result in a change to any part of that name, including the parts that indicate the directory in which the file exists. See? Nothing odd about using the same syntax to rename files as used to move them about at all.

NOTE

Moving or renaming directories is exactly the same as moving or renaming files, with the exception of trying to move a directory from physical media that belongs to one disk partition or network device to media belonging to another. If you try to do this, the system warns you that you're trying to move a directory across partition boundaries and disallows the action. There's no simple way around this, so we'll cover a trick that you can use to get around this limitation of the filesystem in the section on tar later in the chapter. The obvious, and somewhat brute-force solution, is to do it the way it's always been done on the Mac: Copy the directory to the location on the other drive or partition and then delete the original. But doing so changes the modification times of the files, and you might not want that.

CAUTION

You can't create a directory structure by the action of a move command. If you try to move /usr/local/wizbot to /usr/remote/wizbot, and the directory /usr/remote/ does not exist, the mv command exits and notifies you of the error.

Table 10.6 lists the syntax and primary options for mv.

Table 10.6. The Syntax and Primary Options for mv

mv	Moves files.
mv [-f|-i|-n] [-v] <source> <target>
mv [-f|-i|-n] [-v] <source1> <source2> <source3> ... <directory>
In the first form, mv renames <source> to the name provided by <target>. If <source> is a file, a file is renamed. Likewise, if <source> is a directory, a directory is renamed.
In the second form, mv moves the list enumerated by <source1> <source2> <source3> ... to the directory named by <directory>.
-f	Forces an existing file to be overwritten.
-i	Invokes an interactive mode that prompts for a confirmation before overwriting an existing file.

Creating Directories: mkdir

The mkdir command is used to create directories. The usual syntax is simple, being most commonly used as

 mkdir <new directory name> 

This creates a new directory named <new directory name> in the current directory. Full and relative paths are allowed to the new directory being specified. If the path to the directory that you are attempting to create does not completely exist, mkdir does not (by default) create the entire directory structure. For example, if you want to create /usr/local/tmp/testing/morefiles/, and the directory /usr/local/tmp/testing/ does not exist, you either have to create it before you can create /usr/local/tmp/testing/morefiles/ or use the -p option to mkdir. The -p option causes mkdir to create the entire path (if any of it's missing), as well as the final directory. This can be both a great convenience and a real nuisance. If you're an accurate typist, you don't need to issue multiple mkdir commands to create a deep directory hierarchy. On the other hand, because it creates directories that don't exist, a typo in a higher-level directory name just causes the mistyped name to be created, making typing errors annoyingly invisible when they're committed. Users new to Unix might like to add the -v flag as well so that mkdir reports what it's doing as it goes. Table 10.7 shows the syntax and primary options for mkdir.

Table 10.7. The Syntax and Primary Options for mkdir

mkdir	Makes directories.
mkdir [-pv] [-m <mode>] <dir1> <dir2> ...
mkdir creates the named directories in the order specified. The permissions on the directories are controlled by the current umask.
The user must have write permission in the parent directory.
-p	Creates all nonexistent parent directories first. If this option is not specified, the full path prefix of each intended target directory must already exist. Intermediate directories are created with permission bits rwxrwxrwx (0777) as modified by the current umask (2), plus write and execute permission for the owner.
-m <mode>	Sets the permission bits of the created directory to <mode>. <mode> can be in any formats specified to the chmod (1) utility. If a symbolic mode is specified, the operation characters + and - are interpreted relative to an initial mode of a=rwx.
-v	Be verbose about what mkdir is doing.

Copying Files: cp

The cp (copy) command functions similarly to the mv command, but instead of renaming a file between two locations, the cp command creates a duplicate of the file. The syntax is also similar, copying a file from one location to another or copying a number of files into a directory.

If our user nermal wants to copy some images to the /Users/shared directory so that all users can conveniently find the images without having to remember which user has the images, she can copy them with the cp command to the desired location.

Because nermal is new to Unix, she first decides to double-check that the images are located where she thinks they ought to be with the following:

 brezup:nermal Documents $ ls -l Public/Drop\ Box/shar*tiff -rw-r--r--  1 nermal  staff  872714 Apr 16 16:17 Public/Drop Box/sharing-1.tiff -rw-r--r--  1 nermal  staff  873174 Apr 16 16:17 Public/Drop Box/sharing-2.tiff 

Then she actually copies them and verifies that they copied:

 brezup:nermal Documents $ cp Public/Drop\ Box/shar*tiff /Users/shared/ brezup:nermal Documents $ ls -l /Users/shared total 3424 -rw-r--r--  1 nermal  wheel  872714 Apr 23 11:08 sharing-1.tiff -rw-r--r--  1 nermal  wheel  873174 Apr 23 11:08 sharing-2.tiff 

She could have copied each file individually, but cp can fortunately take multiple files in its arguments, so she can use shar*tiff to refer to both files. Note that the copies in the /Users/shared directory show the date that they were copied, rather than the original date. If everyone knows that the images from April 16 are the ones they need to use, they might be confused by the April 23 date. nermal could remove this confusion by specifying the -p option, which preserves as much as possible of the original modification time, user information, and so on:

 brezup:nermal Documents $ cp -p Public/Drop\ Box/shar*tiff /Users/shared/ brezup:nermal Documents $ ls -l /Users/shared total 3424 -rw-r--r--  1 nermal  staff  872714 Apr 16 16:17 sharing-1.tiff -rw-r--r--  1 nermal  staff  873174 Apr 16 16:17 sharing-2.tiff 

Note the use of the \character to escape the space in the folder named Drop Box so that the shell doesn't interpret Public/Drop as one argument and Box/shar*.tiff as another.

If user joray has promised nermal some test data in a subdirectory named tests-for-nermal in joray's home directory, nermal can recursively copy the test directory to her own home directory as follows:

 brezup:nermal Documents $ cp -R ~joray/tests-for-nermal ./ 

A check shows that a directory was indeed copied:

 brezup:nermal Documents $ ls -ld tests-for-nermal drwxr-xr-x  9 nermal  staff  262 Apr 23 11:29 tests-for-nermal 

Not only that, but there was a directory under that directory, and cp copied it:

 brezup:nermal Documents $ ls -l tests-for-nermal total 48 -rw-r--r--  1 nermal  staff   15 Apr 23 11:29 broken -rw-r--r--  1 nermal  staff   20 Apr 23 11:29 broken-again -rw-r--r--  1 nermal  staff   23 Apr 23 11:29 fix -rw-r--r--  1 nermal  staff   17 Apr 23 11:29 fix2 -rw-r--r--  1 nermal  staff  848 Apr 23 11:29 test drwxr-xr-x  4 nermal  staff   92 Apr 23 11:29 test-data -rw-r--r--  1 nermal  staff  848 Apr 23 11:29 test2 

You have seen just some of what you can do with cp. The complete syntax and options for cp are shown in Table 10.8, the command documentation table.

Table 10.8. The Command Documentation Table for cp

cp	Copies files.
cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] <source> <target>
cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] <source1> <source2> ... <directory>
In its first form, cp copies the contents of <source> to <target>.
In its second form, cp copies the contents of the list enumerated by <source1> <source2> ... to the directory named by <directory>. The names of the files themselves are not changed. If cp detects an attempt to copy to itself, that attempt fails.
-R	If <source> is a directory, cp recursively copies the directory. This option also causes symbolic links to be copied as links, rather than indirected through. Created directories have the same mode as the corresponding source directory.
-H	If -R is specified, symbolic links on the command line are followed, but symbolic links in the tree traversal are not.
-L	If -R is specified, all symbolic links are followed.
-P	If -R is specified, no symbolic links are followed.
-f	Forces an existing file to be overwritten.
-i	Invokes an interactive mode that prompts for a confirmation before overwriting an existing file.
-n	Do not overwrite any existing file.
-p	Causes cp to retain as much of the modification time, access time, file flags, file mode, user ID, and group ID information as permissions allow.
-v	Be verbose about the copy process and the results.

NOTE

Apple has added a bit of Apple flavor to a number of Unix command-line utilities and has created a few of its own that merge the world of Apple's GUI and HFS+ filesystem with traditional Unix command-line functionality. If you've a need to copy files with resource forks (such as Classic Mac OS applications and similar files), see Chapter 29, "Maintaining a Healthy System," for documentation on the ditto command. If you've installed the developer tools, and have added the path /Developer/Tools/ to your path, you also have access to CpMac and MvMac, which are Mac-resource-fork-aware versions of cp and mv, as well as a number of other resource-fork-related command-line tools.

Creating Symbolic Links: ln

Sometimes it is useful to link an alternative name with a particular file or directory, or make a file appear to be in two places at once, without actually creating a duplicate of that item. You're already used to this idea from the Finder notion of aliases. At the command line, the same thing can be done with the ln command. It is especially useful for administrative purposes, but even a regular user might need to link a filename or directory name to some other particular name. The best way to do this is to use a symbolic link. The simplest syntax for making a symbolic link is

 ln -s <source> <target> 

Because the syntax is similar to the basic cp syntax, you won't have much trouble remembering it.

You might be wondering just what sort of use you might have for symbolic links. An instance in which you might use a symbolic link is for your website. Suppose that you are using a web editing suite that uses home.html for the default name for the main page of your website. Suppose, however, that your web server is set to read only files named index.html as the default page for a directory. If you use home.html, you might have to give out something like http://ryoohki.biosci.ohio-state.edu/~nermal/home.html as your URL. If, on the other hand, your home.html file were really called index.html, you could give out a slightly shorter URL instead: http://ryoohki.biosci.ohio-state.edu/~nermal/.

What could you do to get an index.html file in your directory if your web editing suite won't create one? And how could you keep it conveniently updated to match home.html? Well, as you just saw, you could copy home.html to index.html. But the next time you edited home.html, you would then have to remember to copy home.html to index.html when you were finished. Although you might be good about remembering to do that, you undoubtedly will eventually forget, probably when it matters most. If you simply link index.html to home.html, every time you update home.html, you don't have to remember to do anything else! To create the symbolic link, do the following:

 brezup:nermal public_html $ ln -s home.html index.html 

A quick check shows us that index.html is now a link to home.html.

 brezup:nermal public_html $ ls -l total 16 -rw-r--r--  1 nermal  staff  52 Apr 23 11:56 home.html lrwxr-xr-x  1 nermal  staff   9 Apr 23 11:56 index.html -> home.html 

NOTE

You will need to set your Apache web server followsymlinks option for symbolic links to work properly as shown. See Chapter 23, "Creating a Web Server," for information on how to do this.

A common administrative use for symbolic links is to move a directory from one partition to another, while leaving the path that a user would use to get to the directory the same. This makes the administrator's change transparent to the user for most purposes. An example of this follows. On this machine, obup, the /usr partition was getting full, so the local directory of /usr/local was moved to a partition named /home. After the directory was moved, the /usr/local directory was replaced with a symbolic link that points to /home/local.

 obup:joray / $ ls -l /usr/local lrwxrwxrwx   1 root     other    11 Mar 20  1997 /usr/local -> /home/local 

If a user changes to the /usr/local directory and then checks his location with pwd, he finds the following:

 obup:joray / $ cd /usr/local obup:joray local $ pwd /home/local 

So, as the user might have grown to expect, he can still cd to /usr/local but without having to know any of the "administrivia" behind its actual location.

Table 10.9 shows the syntax and primary options for ln.

Table 10.9. The Syntax and Primary Options for ln

ln	Makes links.
ln [-fhinsv] <source> <target>
ln [-fhinsv] <source1> <source2> <source3> ... <directory>
In the first form, ln links <source> to <target>. If <target> is a directory, a link named <source> is placed in <target>.
In the second form, ln makes links to the files enumerated by <source1> <source2> <source3> ... in <directory>. The links have the same names as the sources in the list.
There are two types of links: hard links and symbolic links. The default is hard links. A hard link to a file is indistinguishable from the original entry. Hard links may not normally refer to directories and may not span filesystems. Although infrequently necessary, hard link properties make them ideal for creating what are essentially duplicate "real" names for a given file, where the properties of an "alias-like" name are not sufficient. The consequences of this are subtle; for example, with a hard link, you can delete the original file, and it will still exist and take up disk space under its linked name. Because of this, we recommend using soft links unless you're positive that a hard link is necessary for your needs.
A symbolic link refers by name to the file to which it is linked. Symbolic links may refer to directories and may span filesystems.
-f	Forces the link to occur by unlinking any already existing links.
-h	If <target> or <directory> is a symbolic link, it is not followed.
-i	If the target already exists, prompt the user via the error-reporting mechanism (STDERR) regarding the action to take.
-s	Creates a symbolic link this is most like the idea of aliases, with which you're already familiar.

Changing Modification Times and Creating Empty Files: touch

The touch command is used to update the last-modified time for a file. This command also has the side effect of creating a new, empty file if the file that you attempt to touch does not exist.

Neither of these functionalities probably sounds particularly interesting at the outset, but, in fact, they both have good uses. For example, most archiving and backup software is frequently configured to back up only files that have changed since the last backup. Using touch on a file makes it appear to have been changed, which results in it being flagged for backup. Because the touch command can be quickly applied from the command line to a large number of files, it enables you to conveniently force some files to be backed up without having to open and resave each in its parent applications.

Creating empty files doesn't have much use at the command line but turns out to be useful when you start writing programs in the shell scripting language (see Chapter 15). In this case, multiple simultaneously running scripts can be made to talk to each other by the creation of empty files known as flag files essentially the electronic equivalent of a script raising a flag to tell another script that something has happened.

To create a new (empty) file with touch, or to update the modification date of the file to the current time, the syntax for the touch command is simply

 touch <filename to modify> 

The touch command can actually update files to have any modification time that you want, although the preceding is by far the most common usage. Table 10.10 shows the syntax and options for touch.

Table 10.10. The Command Documentation Table for touch

touch	Changes file access and modification times.
touch [-acfm] [-r <file>] [-t [[CC]YY]MMDDhhmm[.SS]] <file> ...
touch sets modification and access times of files to the current time of day. If the file does not exist, it is created with default permissions.
-a	Changes the last-access time of the file, rather than the last-modification time.
-c	Does not create the file if it does not exist.
-f	Attempts to force the update, even if file permissions do not currently permit it.
-m	Changes the modification time of the file.
-r <file>	Replaces access and modification time with that of <file> instead of using the current time.
-t	Changes the access and modification time to the specified time.
CC is the century (20, for dates in the 2000s), YY is the last two digits of the year (2005 can be specified as either 2005 or 05).
MM is the month, DD is the day of the month. hh is the 24-hour version of the hours, mm is minutes, and SS is seconds.
Therefore, to set the date and time of the last change of file fizbin to have occurred on March 2nd, 2002, at 13 seconds after 1:10 AM, you could specify your touch command as touch -t 200203020110.13 fizbin. Alternatively, touch -t 0203020110.13 fizbin would also do.

Removing Files and Directories: rm, rmdir

Removing files and removing directories with Unix are fairly straightforward tasks, so you need to know only two commands to accomplish them.

Now that you have filled up your directory with many files, it is time to learn how to clean it up a bit. You can remove a file with rm. Our user nermal has decided that her file called myfile is no longer needed. To remove it, she does the following:

 brezup:nermal Documents $ ls -l myfile -rw-r--r--  1 nermal  staff  51 Apr 12 15:11 myfile brezup:nermal Documents $ rm myfile remove myfile? Y 

As you notice in this example, rm prompts for confirmation before removing the file. Your system might not be configured to make rm prompt you for the removal of files. You can cause rm to ask for confirmation by aliasing rm to rm -i. Aliasing is a technique covered in more depth in Chapter 15. For now, you want to add either the first or second lines that follow to the .bashrc file or .cshrc file in your home directory (the first if you're using bash, and the second if you're using tcsh). Alternatively, you could add them to /etc/bashrc or /etc/csh.cshrc to make the change for all users of your system.

For bash:

 alias rm='rm -i' 

For tcsh:

 alias rm 'rm -i' 

We highly recommend that you follow these instructions to make your rm command interactive at the first possible opportunity. After you're a seasoned Unix user, you're welcome to run with rm in noninteractive mode by default; but until you've been sure that you're ready a long, long time, it's probably safest to have it operate interactively by default.

CAUTION

Okay, so perhaps suggesting that you put something like a few years of Unix experience under your belt before using rm in the noninteractive mode is a bit facetious and silly. On the other hand, I don't think I know a single longtime Unix user who hasn't entered an rm command, pressed the Return key, and then felt his stomach bounce off the floor as he realized that the reason the command didn't return immediately was because it was currently deleting his entire filesystem. I've seen this happen to seasoned system administrators, so I really do want to stress that with rm, overconfidence can be deadly.

Our user nermal has also decided that she is finished with the data that she copied from user joray's directory and that she wants to remove the entire directory. The easiest way to do this is to force rm to recursively remove the directory, using options -r for recursive, and -f to override the interactive mode she has enabled by default, as shown here:

 brezup:nermal Documents $ ls -ld tests-for-nermal drwxr-xr-x  9 nermal  staff  262 Apr 23 11:29 tests-for-nermal brezup:nermal Documents $ rm -rf tests-for-nermal [localhost:~] nermal% ls -ld tests-for-nermal ls: tests-for-nermal: No such file or directory 

Removing a directory and all its contents using the recursive and force options to rm is easy, but it is also silent and very fast. Remember to use those options only when you really mean it. Double-check everything before you run it. As you can see in the example, there is no recourse if you type the wrong thing.

Table 10.11 shows the syntax and primary options for rm.

Table 10.11. The Syntax and Primary Options for rm

rm	Removes directory entries.
rm [-dfiPRrvW] <file1> <file2> ...
unlink <file>
-f	Forces the removal of files without prompting the user for confirmation. The -f option overrides any previous -i options.
-i	Invokes an interactive mode that prompts for confirmation before removing a file. The -i option overrides any previous -f options.
-d	Attempts to remove directories as well as other types of files.
-R	Attempts to recursively remove files. Implies -d option.
-W	Attempts to undelete files. This option can be used to recover only files covered by whiteouts.
rm removes symbolic links, but not the files referenced by the target of a link. Attempting to remove the files (directories) . and .. is an error because they are virtual placeholders for the current and parent directories, respectively. You can, however, remove the directory that you're currently in by using other valid names for it. If you do this, you'll find that many shell commands suddenly don't work, and produce only an error such as can't stat .. You can usually rescue the situation by issuing a cd ~/ command.

TIP

unlink is a synonym for rm, but it accepts fewer options. This might seem like a bizarre thing to want, but it actually is sometimes an important feature to not have rm's flag options. For example, if you've managed to create a file with a name that starts with a -, using rm to remove it is all but impossible because rm insists on interpreting the - as a marker indicating the presence of a flag, rather than as part of the name. unlink doesn't suffer from this problem, and can let you delete files that have names that confuse rm.

There is also a command available for removing directories: rmdir. It is useful only for removing empty directories, but this is also a valuable limitation on its capabilities. It lets you rmdir rather indiscriminately, and know for certain that it will delete only unoccupied directories, and never any real files. Table 10.12 shows its complete syntax and options.

Table 10.12. The Command Documentation Table for rmdir

rmdir	Removes directories.
rmdir [-p] <directory1> <directory2> ...
rmdir removes each <directory> argument specified, provided it is empty. Arguments are processed in the order listed on the command line. To remove a parent directory and subdirectories of the parent directory, the subdirectories must be listed first.
-p	Attempts to remove the specified directory and its parent directories, if they are empty.