Piping and Directing Output

It's not uncommon for a directory listing or output from another command to scroll off the screen. When using a GUI program like Konsole, you can use the scroll bars to view the output, but what if you are working at the bare command-line prompt?

By pressing Shift+Page Up and Shift+Page Down, you can "scroll" the window up to take a look at some of the old output, but very little is cached in this way and you won't see more than a few screenfuls. A far better solution is to pipe the output of the directory listing into a text viewer like less. Another useful technique is to redirect output to a file.

Piping the Output of Commands

Piping was one of the original innovations provided by Unix. It simply means that you can pass the output of one command to another, which is to say the output of one command can be used as input for another.

This is possible because shell commands work like machines. They usually take input from the keyboard (referred to technically as standard input) and, when they've done their job, usually show their output on the screen (known as standard output).

The commands don't need to take input from the keyboard, and they don't need to output to the screen. Piping is the process of diverting the output before it reaches the screen and passing it to another command for further processing.

Let's assume that you have a directory that is packed full of files. You want to do a long directory listing (ls –l) to see what permissions various files have. But doing this causes reams of output that fly off the screen. Typing something like the following provides a solution:

ls –l | less

The | symbol between the two commands is the pipe. It can be found on most US English-based computer keyboards next to the square bracket keys (above the Enter key; you'll need to hold down the Shift key to get it).

What happens in the example is that ls –l is run by the shell, but rather than sending the output to the screen, the pipe symbol (|) tells BASH to send it to the command that follows, to less. In other words, the listing is displayed within less, where you can read it at leisure. You can use Page Up and Page Down or the arrow keys to scroll through it. Once you quit less, the listing evaporates into thin air; the piped output is never actually stored as a file.

In the previous section, you saw how you can use the history command to view the command history. At around 1,000 entries, its output scrolls off the screen in seconds. However, you can pipe it to less, like so:

history | less

Figure 18-3 shows the result on my test PC.

Figure 18-3. Piping the output of the history command into the less command lets you read the output fully.

You can pipe the output of any command. One of the most common uses is when searching for a particular string in the output of a command. For example, let's say you know that, within a crowded directory, there's a file with a picture of some flowers. You know that the word flower is in the filename, but can't recall any other details. One solution is to perform a directory listing, and then pipe the results to grep, which is able to search through text for a user-defined string (see Chapter 16):

ls –l | grep –i 'flower'

In this example, the shell runs the ls –l command, and then passes the output to grep. The grep command then searches the output for the word flower (the –i option tells it to ignore uppercase and lowercase). If grep finds any results, it will show them on your screen.

The key point to remember is that grep is used here as it normally is at the command prompt. The only difference is that it's being passed input from a previous command, rather than being used on its own.

You can pipe more than once on a command line. Suppose you know that the filename of the picture you want involves the words flower and daffodil, yet you're unsure of where they might fall in the filename. In this case, you could type:

ls –l | grep –i flower | grep -i daffodil

This will pass the result of the directory listing to the first grep, which will search the output for the word flower. The second pipe causes the output from grep to be passed on to the second grep command, where it’s then searched for the word daffodil. Any results are then displayed on your screen.

Redirecting Output

Alongside piping is the concept of redirecting. This is like piping, except that the output is passed to a file rather than to another command. Redirecting can also work the other way: the contents of a file can be passed to a command.

If you wanted to create a file that contained a directory listing, you could type:

ls –l > directorylisting.txt

The angle bracket (>) between the commands tells BASH to direct the output of the ls –l command into a file called directorylisting.txt. If a file with this name exists, it's overwritten with new data. If it doesn't exist, it's created from scratch.

You can add data to an already existing file using two angle brackets:

ls –l >> directorylisting.txt

This will append the result of the directory listing to the end of the file directorylisting.txt, although, once again, if the file doesn't exist, it will be created from scratch.

Redirecting output can get very sophisticated and useful. Take a look at the following:

cat myfile1.txt myfile2.txt > myfile3.txt

As you learned in Chapter 16, the cat command joins two or more files together. If the command were used on its own without the redirection, it would cause BASH to print myfile1.txt on the screen, immediately followed by myfile2.txt. As far as BASH is concerned, it has joined myfile1.txt to myfile2.txt, and then sent them to standard output (the screen). By specifying a redirection, you have BASH send the output to a third file. Using cat with redirection is a handy way of combining two files.

It's also possible to direct the contents of a file back into a command. Take a look at the following:

sort < textfile.txt > sortedtext.txt

The sort command simply sorts words into alphanumeric order (it actually sorts them according to the ASCII table of characters, which places symbols and numbers before alphabetic characters). Directly after the sort command is a left angle bracket, which directs the contents of the file specified immediately after the bracket into the sort command. This is followed by a right angle bracket, which directs the output of the command into another file.

Tip 

To see a table of the ASCII characters, type man 7 ascii at the command-line prompt.

There aren't many instances in day-to-day usage where you'll want to use the left angle bracket. It's mostly used with the text-based mail program (which lets you send e-mail from the shell), and in shell scripting, in which a lot of commands are combined together to form a simple program.

Redirecting Standard Error Output

Standard input and standard output are what BASH calls your keyboard and screen. These are the default input and output methods that programs use unless you specify something else, such as redirecting or piping output and input.

When a program goes wrong, its error message doesn't usually form part of standard output. Instead, it is output via standard error. Like standard output, this usually appears on screen.

Sometimes, it's very beneficial to capture an error message in a text file. This can be done by redirecting the standard error output. The technique is very similar to redirecting standard output:

cdrecord --scanbus 2> errormessage.txt

The cdrecord command is used to burn CDs, and with the --scanbus command option, you tell it to search for CD-R/RW drives on the system, something which frequently results in an error message if your system is not properly configured.

After the initial command, you see the redirection. To redirect standard error, all you need to do is type 2>, rather than simply >. This effectively tells BASH to use the second type of output: standard error.

You can direct both standard output and standard error to the same file. This is done in the following way:

cdrecord --scanbus > error.txt 2>&1

This is a little more complicated. The standard output from cdrecord --scanbus is sent to the file error.txt. The second redirect tells BASH to include standard error in the standard output. In other words, it's not a case of standard output being written to a file, and then standard error being added to it. Instead, the standard error is added to standard output by BASH, and then this is written to a file.