Pipes and Piping

Sometimes the thing that makes the most sense is to feed the output from one command directly into another command without having to resort to files in between at every step of the way. This is called piping. The symbolism is not that subtle: Imagine pieces of pipe connecting one command with another. Not until you run out of pipe does the command's output emerge. The pipe symbol is the broken vertical bar on your keyboard, usually located just below or (depending on the keyboard) just above the Enter key and sharing space with the backslash key. Here's how it works:

 cat random_names | sort | wc  w > num_names 

In this example, the output from the cat command is piped into sort, whose output is then piped into the wc command (that's "word count"). The -w flag tells wc to count the number of words in random_names. So far, so good.

That cat at the beginning is actually redundant, but I wanted to stack up a few commands for you to give you an idea of the power of piping. Ordinarily, I would write that command as follows:

 sort random_names | wc  w > num_names 

The cat is extraneous because sort incorporates its function. Using pipes is a great timesaver because you don't always need to have output every step of the way.

tee: A Very Special Pipe

Suppose you want to send the output of a command to another command, but you also want to see the results at some point. Using the previous word count example, if you want a sorted list of names but also want the word count, you might have to use two different commands: one to generate the sorted list and another to count the number of words. Wouldn't it be nice if you could direct part of the output one way and have the rest continue in another direction? For this, use the tee command:

 sort random_names | tee sorted_list | wc  w > num_names 

The output from sort is now sitting in a file called sorted_list, while the rest of the output continues on to wc for a word count.

STDERR

What about STDERR? Some commands (many, in fact) treat the error output differently than the STDOUT. If you are running the command at your terminal and that's all you want, there is no problem. Sometimes, though, the output is quite wordy and you need to capture it and look at it later. Unfortunately, using the STDOUT redirect (the greater-than sign) is only going to be so useful. Error messages that might be generated (such as warning messages from a compilation) will go to the terminal, as before. One way to deal with this is to start by redirecting STDERR to STDOUT and then to redirect that to a file. Here's the line I use for this:

 command_name 2>&1 > logfile.out 

Remember that file descriptor 2 is STDERR and that file descriptor 1 is STDOUT. That's what that 2>&1 construct is all about. You are redirecting fd2 to fd1 and then redirecting that output to the file of your choice. Using that program compilation example, you might wind up with something like this:

 make  f Makefile.linux 2>&1 > compilation.output 

Quick Tip

The final greater-than sign in the preceding example could be eliminated completely. When using the 2>&1 construct, it is assumed that what follows is a filename.

The Road to Nowhere

If the command happens to be verbose by nature and doesn't have a quiet switch, you can redirect that STDOUT and STDERR noise to what longtime Linux users like to call the bit bucket, a special file called /dev/null literally, a road to nowhere. Anything fed to the bit bucket takes up no space and is never seen or heard from again. When I was in school, we would tell people to shut up by saying, "Dev null it, will you?" As you can see, we were easily amused.

To redirect output to the bit bucket, use the STDOUT redirection:

 command  option > /dev/null 

If, for some strange reason, you want to sort the output of the random_names files and you do not want to see the output, you can redirect the whole thing to /dev/null in this way:

 sort random_names > /dev/null 

Using the program compilation example where you had separate STDOUT and STDERR streams, you can combine the output to the bit bucket:

 make  F makefile.linux 2>&1 /dev/null 

That's actually a crazy example because you do want to see what goes on, but redirecting both STDOUT and STDERR to /dev/null is quite common when dealing with automated processes running in the background.