Not all text data is suitable for storage in a relational database table. For example, letters and memos do not fit the model of rows and columns . A file that is processed as a sequential list of bytes, such as a letter or memo, is known as a stream file. Stream files are widely used in business.
Stream files are not organized into rows (records) and columns (fields). A stream-file record is a series of characters that ends with a special code, such as a carriage -return character followed by a line-feed character. Stream-file records may vary in length. One record of a file may be 20 bytes long, the next 150, and a third 12. Stream files may contain binary data (such as sounds or pictures) or text data.
Qshell is well suited to working with stream files. Most Qshell commands read a stream of characters and write a stream of characters.
Qshell defines three standard stream files by default. You do not have to declare, open , or close these files. The default input file is known as standard input and is abbreviated to stdin . It is assigned to the keyboard, but you may use redirection to assign it to another file, or a pipeline to read the output of another Qshell script or utility.
There are two predeclared output files: standard output ( stdout ) and standard error ( stderr ). The first is for the normal output of commands, while the second one is for error messages. Both files are directed to the display by default when Qshell runs interactively, but may be redirected to disk files or piped to other Qshell utilities. The fact that there are two output files allows you to separate good output from error output.
The Qshell interpreter is a good example of the use of stream files. In an interactive Qshell session, the input to Qshell is the keyboard, and the output device is the display. The session is a conversation. Each time you press the Enter key, Qshell reads the stream of characters you typed, attempts to carry out your instructions, and gives you a stream of characters as output, even if that output is only a prompt.
A terminal session is shown in Figure 10.1. It shows a user typing the List Directory Contents command ( ls ) to determine what .csv files were in the current directory. Qshell reads the command from stdin and sends the list of file names to stdout. The user then types the Remove Directory Entries command ( rm ) to delete a file. Qshell reads the command from stdin and responds by sending a confirmation message to stderr. Qshell reads the user's response from stdin. And so the dialog continues.
/home/JSMITH $ ls *.csv cust.csv two.csv uuu.csv /home/JSMITH $ rm -i uuu.csv rm: 001-2145 Do you want to remove the file or directory uuu.csv (Y or N)? y /home/JSMITH $ ls *.csv cust.csv two.csv /home/JSMITH $
Figure 10.1: By default, Qshell utilities use the keyboard and display devices for input.
Stdin, stdout , and stderr may be assigned to devices through a process known as redirection, which is somewhat like an override in OS/400. Qshell supports nine redirection operators, shown in Table 10.1. (The "here" document and noclobber option are explained later in this chapter.)
Operator |
Description |
---|---|
< |
Redirect input |
<& |
Duplicate input |
<&- |
Close input |
<<, <<- |
Open a "here" document |
> |
Replace output |
> |
Replace output; ignore the noclobber option |
>> |
Append output |
>& |
Duplicate output |
>&- |
Close output |
The most commonly used redirection operators are <, >, and >>. They are illustrated in this section. The others will be explained when file descriptors are discussed.
The redirection operators may be separated from the file names that follow them, but it is not necessary. In other words, the following two commands are equivalent:
ls *.qsh >temp.txt ls *.qsh > temp.txt
To illustrate the use of redirection operators, the input data in Figure 10.2 is used as the basis of the examples that follow.
cat goodoleboys.txt Name Born Phone Dog Wife Shotgun Paid ========= ======== ======== ======== ========= ======= ===== Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy BobJune 11 444-4340 Leotis Lisa Sue 12 Amos Jan 4 333-1119 Amos Abigail 20 Otis Sept 17 444-8000 Ol' Sal Sally 12 Claude May 31 333-4340 Blue Etheline 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Junior April 30 BR-549 Percival Lilly Faye 12 Bill Feb 29 333-4444 Daisy Daisy 20 Ernest T.?? none none none none
Figure 10.2: The goodoleboys.txt file is used in this chapter's examples.
Figures 10.3 through 10.6 use the Translate Characters utility ( tr ), which can translate characters in one character set to corresponding characters in another character set. Tr works well for these examples because it reads from stdin and writes to stdout. In Figure 10.3, the redirection operator causes tr to read the goodoleboys.txt file, not the keyboard.
tr [:lower:] [:upper:] NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======== ========= ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE
Figure 10.3: Lowercase letters are translated to their uppercase equivalents. Output goes to the display because stdout is not redirected.
Figure 10.4 also read the goodoleboys.txt file and translates lowercase letters to their uppercase equivalents. In this case, though, output goes to the file upper.txt, erasing any existing data in it. If the file does not exist, it is created.
tr [:lower:] [:upper:] upper.txt /home/JSMITH $ cat upper.txt NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======= ======== ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE /home/JSMITH $
Figure 10.4: The > operator replaces data in file upper.txt.
In Figure 10.5, the translated data from the goodoleboys.txt file is appended to the upper.txt file, so its existing two records are not overwritten. Again, if the file did not exist, it would be created.
cat upper.txt I LIKE CHEESE! + + + + + /home/JSMITH $ tr [:lower:] [:upper:] >upper.txt /home/JSMITH $ cat upper.txt I LIKE CHEESE! + + + + + NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======== ========= ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE /home/JSMITH $
Figure 10.5: Because of the >> operator, new data is appended to the output file.
The tr command in Figure 10.6 translates the uppercase letters in source member MYPGM of file MYLIB/QRPGLESRC to their lowercase equivalents. Other characters are unaffected. The output is placed in source member MYPGM2 in the same source physical file.
cat /qsys.lib/mylib.lib/qrpglesrc.file/mypgm.mbr D CD S 35 DIM(3) CTDATA PERRCD(1) C *ENTRY PLIST C PARM INDEX 1 0 C PARM DESCR 35 * C INDEX IFLT 1 C INDEX ORGT 35 C MOVE *ALL'X' DESCR C ELSE C MOVE CD(INDEX) DESCR C ENDIF * C RETURN /home/smith $ tr [:upper:] [:lower:] >/qsys.lib/mylib.lib/qrpglesrc.file/mypgm2.mbr /home/smith $ cat /qsys.lib/mylib.lib/qrpglesrc.file/mypgm2.mbr d cd s 35 dim(3) ctdata perrcd(1) c *entry plist c parm index 1 0 c parm descr 35 * c index iflt 1 c index orgt 35 c move *all'x' descr c else c move cd(index) descr c endif * c return
Figure 10.6: File redirection can be used with database file members .
If the > operator is used alone (that is, it does not follow a command), the effect is to clear the file. Consider this example:
>upper.txt /home/JSMITH $ cat upper.txt /home/JSMITH $
The redirection operator is used without a command, so it clears upper.txt. This is the Qshell version of CL's Clear Physical File Member command (CLRPFM).
A word of warning is in order:
Do not use the same file in both input and redirection operators.
Here is an example of what you should not do:
# WARNING: Do NOT do the following !!!! tr [:upper:] [:lower:] < myfile >myfile # Do NOT do this !!!! # WARNING: Do NOT do the preceding !!!!
The lowercase version of myfile will not replace the uppercase version. Redirection is a function of Qshell, not of tr . Therefore, the output redirection operator will clear myfile before tr begins to read the data.
By default, the > redirection overwrites the contents of an existing file. In Figure 10.7, output.txt is loaded with a list of the Qshell script files (the files with an extension of .qsh). Then, the file is replaced with a list of the names of .csv files.
ls *.qsh >output.txt /home/JSMITH $ ls *.csv >output.txt /home/JSMITH $ cat output.txt cust.csv two.csv uuu.csv /home/JSMITH $
Figure 10.7: The output of the second ls command replaces , or "clobbers," the output of the first ls command.
To prevent the > redirection operator from overwriting an existing file, use the "noclobber" option, which is turned on using the set -C command. Figure 10.8 illustrates the use of this command to prevent Qshell from overwriting file output.txt. When the script attempts to replace the contents of output.txt with a list of the names of .csv files, Qshell responds with error message 001-0054. The cat command shows that the second list did not change the contents of output.txt.
set -C /home/JSMITH $ ls *.qsh >output.txt /home/JSMITH $ ls output.txt output.txt /home/JSMITH $ ls *.csv >output.txt qsh: 001-0054 The noclobber option is set and file output.txt already exists. /home/JSMITH $ cat output.txt args01.qsh args02.qsh args03.qsh /home/JSMITH $
Figure 10.8: Use the noclobber option to prevent accidental erasure of data.
There are two ways to turn on the noclobber option:
set -C set -o noclobber
To turn off the noclobber option, use a plus sign instead of a minus sign, like this:
set +C set +o noclobber
You can override the noclobber option on a single operation by using the > redirection operator, as shown in Figure 10.9. The set utility turns on the noclobber option. The first ls fails because output.txt already exists and the > redirection operator is used. The second ls command succeeds because the > redirection operator overrides the noclobber option. Notice that the > operator (in the first ls command) generates an error message, but the > (in the second ls command) does not. The cat command shows that the .csv files are listed in file output.txt.
set -C /home/JSMITH $ ls *.csv >output.txt qsh: 001-0054 The noclobber option is set and file output.txt already exists. /home/JSMITH $ ls *.csv >output.txt /home/JSMITH $ cat output.txt cust.csv two.csv uuu.csv /home/JSMITH $
Figure 10.9: The > redirection operator overrides the noclobber option.
A pipeline is a series of commands connected with the pipe symbol, the vertical bar (). A pipe channels stdout of one command to stdin of another. Figures 10.10 through 10.12 show pipelines in action. Input comes from the goodoleboys.txt file, which was shown in Figure 10.2.
Figure 10.10 shows the use of grep , a search utility, to look in the goodoleboys.txt file for records that include the string 444-. Instead of writing the selected records to stdout, it sends them into the pipe. The sort routine reads from stdin, so it gets the selected data from the pipe, sorts it, and sends it along to stdout. Since stdout is not piped or redirected from sort , the sorted data appears on the display.
grep '444-' Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Bubba Oct 13 444-1111 Buck Mary Jean 12 Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410
Figure 10.10: Grep is a search utility. It looks for records that match a pattern.
Figure 10.11 uses the grep utility with two pipes. Grep selects the records that contain 444- and sends them to tr , which converts the lowercase letters to uppercase, leaving all other characters as they are, and passes the data along to the sort utility. After sorting, the data is sent to stdout, which is the display in this case because there is no output-redirection operator or pipe following the sort.
grep '444-' ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410
Figure 10.11: There are two pipes in this pipeline.
Pipes can also be combined with redirection operators, as Figure 10.12 shows. Grep selects all goodoleboys.txt records that contain 444- and sends them to tr , which converts lowercase letters to uppercase and passes the data along to sort . Sort sends the sorted data to the file sorted.txt. In this case, the sorted records containing 444- are redirected to the file sorted.txt.
> grep '444-' sorted.txt /home/JSMITH $ > cat sorted.txt ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 /home/JSMITH $
Figure 10.12: Pipes can be combined with redirection operators.
It is common for beginning shell users to confuse redirection operators and pipes. There is, however, a great difference between the two concepts. Consider the following two commands:
myscript.qsh > somefile myscript.qsh somefile
In the first line, the output of myscript.qsh will be stored in somefile . In the second command, the output of myscript.qsh is to be read by a program or script called somefile . Since files in Unix systems may contain any type of data, including programs, using an output redirection operator instead of a pipe before a program or script destroys or corrupts the program or script.
That warning is worth repeating:
Using an output redirection operator instead of a pipe before a program or script destroys or corrupts the program or script.
It will help you to use these properly if you always look at the filename to which stdout is being redirected. If the file is a program or script, use a pipe. If it is a data file, use a redirection operator. It will also help if you think of a pipe operator as a symbol for a temporary file. Imagine that the command preceding the pipe symbol writes to a temporary file, and the command following the pipe symbol reads the temporary file.
You might decide that you would like to see the data that travels through a pipe, perhaps for debugging purposes. If so, use the tee utility, also known as duplicate standard input . As you might guess, the tee utility gets its name from the "T" shape a plumber might install in a real pipe to branch its flow between two locations. Tee reads stdin and writes the unchanged data to both stdout and one or more other files. The syntax of tee utility is shown here:
tee [ -ai ] [ file ]
By default, tee replaces the data in a file that already exists. To append the data instead, use the a option. Use of tee is shown in Figure 10.13. It passes the output of grep along to tr , and at the same time makes a copy of the output in the file tee.out. Notice that the data is in mixed case in the tee.out file, because it had not been fed to the tr command at that point. Notice also that the data in tee.out is not sorted, because the sort process had not yet run.
grep '444-' > tr [:lower:] [:upper:] sort >sorted.txt /home/JSMITH $ cat tee.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $
Figure 10.13: After the pipeline finishes, the cat command shows the input that grep was given.
In Figure 10.14, grep again selects records that contain the string 444- from file goodoleboys.txt. Then, tee writes the records to two files, tee.out and two.out, and at the same time passes the records to the sort utility.
grep '444-' Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Bubba Oct 13 444-1111 Buck Mary Jean 12 Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 /home/JSMITH $ cat tee.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $ cat two.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $
Figure 10.14: The tee utility writes the records from grep to tee.out and two.out, and passes the records to sort.
You can temporarily override stdout on a line-by-line basis. That is, even though stdout is assigned to a certain file, you can direct the output of individual commands to other files. Consider this example of redirected output:
print "Something" if [ -z "" ] then date >>errorlog.txt print "Script:
print "Something" if [ -z "$1" ] then date >>errorlog.txt print "Script: $0" >> errorlog.txt print "Parm 1 missing, default used." >> errorlog.txt fi print "Something else"
" >> errorlog.txt print "Parm 1 missing, default used." >> errorlog.txt fi print "Something else"
The first and last print commands write to stdout. The output from those commands will go to the display, to a file, or into a pipe, depending on how stdout is directed. Look at the if block, however. Three commands ” date , and two echo commands ”do not write to stdout, but append to errorlog.txt. Figure 10.15 shows what happens when this code is executed. When the user types the name of the script, error01.qsh, on the command line, Qshell runs the script. Since stdout is not redirected, the first and last print commands write the string "Something" and "Something else" to the terminal. The cat utility shows what has been written to file errorlog.txt.
error01.qsh Something Something else /home/JSMITH $ cat errorlog.txt Wed Dec 25 08:27:35 2002 Script: /home/JSMITH/error01.qsh Parm 1 missing, default used. /home/JSMITH $
Figure 10.15: The previous script redirected both the stdout and stderr files.
A "here" document is a special way of capturing input lines and directing them as a group to another process or file. The "here" document operator, <<, is followed by an end-of-data delimiter string of the programmer's choosing.
Figure 10.16 shows how to create a file from the keyboard using a "here" document. The user chooses the string EOD to indicate the end of the input data. The cat utility displays the lines to stdout , which is overridden to file mydata.txt.
cat <mydata.txt > Jack,12 > Bill,20 > Leroy,15 > EOD /home/JSMITH $ cat mydata.txt Jack,12 Bill,20 Leroy,15 /home/JSMITH $
Figure 10.16: This example shows how to use a "here" document to enter data into an IFS file from the keyboard.
You may follow the << operator with a hyphen if you wish. This allows you to enter a tab character before the end-of-data delimiter. Since you can't type a tab character from a 5250 terminal session, you are not likely to use this option unless you edit scripts on an ASCII-based machine.
A filter is a program that reads from stdin and writes to stdout . Filters are designed with a single purpose in mind, and are written to be flexible. The sort utility, for example, is a filter that resequences, merges, and sequence-checks text files. Because sort is so powerful, it is unlikely that you will need to write another program to sort.
Generally, you should write scripts and programs that embrace the filter philosophy. As a rule, scripts should not converse with the user. For instance, suppose you want the user to enter his or her name when a script begins to run. You could use print or echo to display a prompt, and use read to accept the user's input, like this:
echo "Please enter your name:" read name
However, this degree of interactivity would make it difficult or impossible to use the script in a pipeline. It is better to provide a way to specify a command line option. In the following example, the user name is entered through a parameter:
myscript.qsh -n 'Joe Smith'
As you can see, the use of switches can replace interactive I/O. In this case, the programmer writes the script so that it interprets the parameter following the n option as a user's name. This is demonstrated in the following example:
if [ "" ] && [ "" ] && [ = "-n" ] then name= else name=$LOGNAME fi
If the first and second positional parameters are defined and the first parameter is -n , the second parameter provides the value for the variable name; otherwise , name gets its value from the Qshell predefined variable LOGNAME, which is the ID of the user who is running the shell. There is no need to use echo and read statements to include the user's name.
To write effective Qshell scripts, you must learn to make Qshell scripts read data from and write data to files. The Qshell I/O utilities, listed here, are streamoriented and easy to use:
While there is only one input utility, there are several Qshell output utilities. The print utility, which writes to output files, is probably the one you'll need most often. Here is its syntax:
print [ -nrR ] [ -u [ n ] ] [ argument ]
Print writes zero or more arguments to stdout . Each argument is separated by a space. Unless you include the -n option, print ends with a newline character, which forces the next output into another record. Print lets you embed control sequences (listed in Table 10.2) in output.
Sequence |
Description |
---|---|
a |
Sounds the terminal's alarm |
|
Backspaces one character |
c |
Ignores subsequent arguments and suppress newline |
f |
Formfeed (clears the screen) |
|
Newline (combines carriage return and linefeed ) |
|
Return (carriage return, but no linefeed; returns to beginning of the line) |
|
Tab |
v |
Vertical tab (linefeed, but no carriage return; moves cursor down) |
\ |
Prints one backslash character |
|