|< Day Day Up >|
Automating Tasks with Shell Scripts
With as many times as we've mentioned how powerful shell scripting can be and how much time and effort it can save you, you might be expecting that writing shell scripts is going to require dealing with some additional level of complexity on top of what you've already learned. Shell scripts are simple programs that you write in the language of the shell, and if you've made it this far in the book, you've been learning and working in the language of the shell for a few chapters now. If you consider this fact, and the notion that Unix, by design, attempts to abstract the notion of input and output so that everything looks the same to the operating system, you might have a good guess at what we'll say next: That's right you already know how to write shell scripts. There are a few more shell techniques that you can learn to enhance your ability to program the shell, but Unix itself doesn't care whether it's you typing at a command prompt or commands being read out of a file on disk. Everything you've typed so far in working with the shell could have been put in a file, and the computer could have typed it to itself voilá, a shell script.
At its most trivial, a shell script can be exactly what you type at a prompt to accomplish some set of tasks. If you find that you have a need to repeatedly execute the same commands, you can type them once into a file, make that file executable, and forever after execute them all just by typing the name of the file.
It really is as simple as it sounds, but just in case it's not quite clear yet, an example should help. Consider the following situation: Let's say that every day when you log in to your computer, you like to check the time (with date), check to see who's online (using the who command), check to see how much space is left on the drive with your home directory (with df), and finally check who's most recently sent you mail (with from - this command might not work for you on a default install, because it relies on my particular mail setup).
You could type each of these things to a command prompt when you log in to your machine, or you could put them in a file, make it executable, and let the file "type" them for you.
brezup:ray Documents 165 $ date Mon Jun 18 23:35:55 EDT 2003 brezup:ray Documents 166 $ who joray ttyp0 Jun 14 18:22 (220.127.116.11) ray ttyp1 Jun 18 21:49 (18.104.22.168) ray ttyp2 Jun 15 10:00 (rodan.chi.ohio-s) radman ttyp3 Jun 18 23:33 (ac9d3e22.ipt.aol) brezup:ray Documents 167 $ df . Filesystem 512-blocks Used Avail Capacity Mounted on /dev/disk0s11 12581856 12224784 357072 97% /Volumes/Wills_Data brezup:ray Documents 168 $ from | tail -10 From firstname.lastname@example.org Mon Jun 18 16:20:23 2003 From email@example.com Mon Jun 18 17:28:33 2003 From firstname.lastname@example.org Mon Jun 18 18:34:28 2003 From email@example.com Mon Jun 18 19:23:42 2003 From firstname.lastname@example.org Mon Jun 18 20:42:53 2003 From email@example.com Mon Jun 18 21:24:00 2003 From firstname.lastname@example.org Mon Jun 18 22:02:15 2003 From email@example.com Mon Jun 18 22:28:56 2003 From firstname.lastname@example.org Mon Jun 18 23:15:28 2003 From email@example.com Mon Jun 18 23:34:43 2003 brezup:ray Documents 169 $ cat > imhere #!/bin/bash date who df . from | tail -10 ^D brezup:ray Documents 170 $ chmod 755 imhere brezup:ray Documents 171 $ ./imhere Mon Jun 18 23:36:51 EDT 2003 joray ttyp0 Jun 14 18:22 (22.214.171.124) ray ttyp1 Jun 18 21:49 (126.96.36.199) ray ttyp2 Jun 15 10:00 (rodan.chi.ohio-s) Filesystem 512-blocks Used Avail Capacity Mounted on /dev/disk0s11 12581856 12224784 357072 97% /Volumes/Wills_Data From firstname.lastname@example.org Mon Jun 18 17:28:33 2003 From email@example.com Mon Jun 18 18:34:28 2003 From firstname.lastname@example.org Mon Jun 18 19:23:42 2003 From email@example.com Mon Jun 18 20:42:53 2003 From firstname.lastname@example.org Mon Jun 18 21:24:00 2003 From email@example.com Mon Jun 18 22:02:15 2003 From firstname.lastname@example.org Mon Jun 18 22:28:56 2003 From email@example.com Mon Jun 18 23:15:28 2003 From firstname.lastname@example.org Mon Jun 18 23:34:43 2003 From email@example.com Mon Jun 18 23:36:48 2003
As you can see, executing the file imhere, containing my commands, produces essentially the same output with much less typing. (The output has a few changes because one user has left the system, and new mail has arrived between the by-hand runs and the execution of the shell script.)
The only part of the imhere script that might be confusing is the first line: #!/bin/bash. The shell interprets the first line of a shell script in a special manner. If a pattern such as #!<path to an executable file> is found, the executable file named in that line is used as the shell for executing the contents of the script.
Single-Line Automation: Combining Commands on the Command Line
Before we go too far with the notion of storing collections of commands in files, however, let's look at what can be done at just the command-line level. You already know about using pipes and variables. These concepts can be combined to produce very powerful expressions directly at the command line, without any need to store the collection of commands in a file.
Consider for a moment the netpbm collection of graphics manipulation programs that was installed in Chapter 14, "Command-Line Software Installation and Troubleshooting." Included in the capabilities of the suite are a number of conversions among various file formats, as well as a range of manipulations of the image content itself. With Mac OS, if you want to convert a GIF file into a PICT file, and convert it to four-color grayscale along the way, you have a number of options. You could fire up PhotoShop or GraphicConverter, perform the changes there, and save the file. Alternatively, you could program a conversion filter in DeBabelizer to perform this manipulation for you. With netpbm, you can perform the manipulation from the command line as shown in the code following the note.
brezup Documents 277> ppmtogif < sage.ppm > sage.gif ppmtogif: computing colormap... ppmtogif: 192 colors found brezup Documents 278> giftopnm < sage.gif > sage.pnm brezup Documents 279> ppmtopgm < sage.pnm > sage.pgm brezup Documents 280> ppmquant 4 < sage.pgm > sage.pgm2 ppmquant: making histogram... ppmquant: 120 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... brezup Documents 281> ppmtopict < sage.pgm2 > sage.pict ppmtopict: computing colormap... ppmtopict: 4 colors found
Figure 15.1 shows a comparison of the original image sage.gif, and the four-color grayscale image, sage.pict.
Figure 15.1. A comparison of an original file and the result of processing it through one of a number of different netpbm filters.
From the brief discussion at the beginning of this section, you should already have an idea of how you could combine all that into a single file, if for some reason you wanted to perform that conversion to the sage.gif file over and over and over.
This does not seem to be a very useful thing to automate, and takes quite a bit of typing to boot (although, frankly, not nearly as much work as starting up Photoshop to do something this simple). Let's see what we can do with pipes and shell variables though to cut down on the amount of typing.
First, observe that all the programs are taking the input files on STDIN and are producing output on STDOUT. Unix command-line programs are frequently like this, and it's a very good thing. Using the power of pipes to connect one program's STDOUT to another program's STDIN, we can shorten that collection of commands to a single command line:
brezup Documents 287> giftopnm < sage.gif | ppmtopgm | ppmquant 4 | ppmtopict > sage.pict ppmquant: making histogram... ppmquant: 120 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... ppmtopict: computing colormap... ppmtopict: 4 colors found
I'll let you verify that the output is graphically identical on a file of your own.
You might think that it's probably not very likely that you'll want to perform this single manipulation repeatedly to the same image. However, there are many times when you'd like to be able to perform a collection of manipulations like that on a number of different images. With what you know about shell variables, you can come up with several ways to abstract that command line so that it could be reused for any GIF file. You might try something like this:
brezup Documents 288> set infile=sage.gif brezup Documents 289> giftopnm < $infile | ppmtopgm | ppmquant 4 | ppmtopict > $infile:r.pict ppmquant: making histogram... ppmquant: 120 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... ppmtopict: computing colormap... ppmtopict: 4 colors found
You could simply use new values for $infile, and you'd have a reusable command that could perform the same manipulation on any GIF image. It's still too much work though, right? Well, remember aliases? We can further automate things by using an alias command to compact that large command-line expression into something more manageable.
brezup Documents 290> alias greyconvert 'set infile=\!#:* ; giftopnm < $infile | ppmtopgm | ppmquant 4 | ppmtopict > $infile:r.pict ' brezup Documents 291> greyconvert sage.gif ppmquant: making histogram... ppmquant: 120 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... ppmtopict: computing colormap... ppmtopict: 4 colors found
That command is getting pretty long, isn't it? The good news is that for almost any task in Unix, you can figure out how to build up to an expression like this, just as shown here. Start by figuring out how to do it one step at a time on the command line, and work your way up to an elegant solution that solves the problem for you with as little repetitive work as necessary.
After you've invented useful aliases or functions such as this one for yourself, remember to store them in your .cshrc, .tcshrc, or .bashrc file in your home directory so that you can use them again whenever you log in to your computer.
Multiline Automation: Looping at the Prompt
Creating customized commands that perform special functions such as the greyconvert command built in the previous sections is useful, but it still doesn't address the need to automate tasks. For that, we need some sort of looping command, and bash and tcsh both offer two basic types of loops: the for loop and the while loop. Both commands repeat a block of shell commands. The first executes it "for each" of its arguments, and the second executes it "while" some condition is true.
Looping in tcsh
The foreach and while shell commands are unlike other shell commands in that they require additional information beyond the first command line. For example, the syntax of the foreach command in tcsh is
foreach <variablename> ( <item list> ) <first command to execute> <second command to execute> . . . <nth command to execute> end
The tcsh while command, on the other hand, has the syntax
while ( <comparison> ) <first command to execute> <second command to execute> . . . <nth command to execute> end
In the foreach command, the <item list> can be a space-separated list of items, a command that produces a space-separated (or return-separated) list of items, or a command-line wildcard that matches a list of files. As a demonstration, consider a situation in which we want to execute our previous greyconvert command on every GIF file in a directory containing many files. This can be accomplished in several ways by the use of the foreach command.
Looping in bash
The syntax for the for command in bash is
for <variablename> in <words> ; do <first command to execute> ; <second command to execute> ; . . . <nth command to execute> ; done
The bash while command, on the other hand, has the syntax
while <commands> ; do <first conditional command to execute> ; <second conditional command to execute> ; . . . <nth conditional command to execute> ; done
In the for command, bash steps throught the list of <words> and sets the value of <variablename> sequentially to each. After <variablename> has been set to any given value from <words>, the list of commands is executed. When the final value has been extracted from <words> and the commands have been run the final time, the for loop terminates.
In the while command, bash executes <commands> (which can be a single word command, or a parenthesized, colon-separated list of commands). If the final command in <commands> returns a TRue (1) value, the while loop executes the conditional commands in the loop. This repeats until the final command in <commands> evaluates to false (0).
Using Looping Constructs at the Command Line
Using looping constructs at the command line is relatively powerful and allows you to execute a collection of commands many times, for many files (again, tcsh examples are somewhat clearer to read). The following example uses a wildcard to match all the files of interest in the current directory, and then runs our greyconvert alias on each of them:
brezup amg 246> ls AMG_cal-cover.gif AhMyGoddess-v05.gif AhMyGoddess-v10-f1.gif AhMyGoddess-v01-f1.gif AhMyGoddess-v06-f1.gif AhMyGoddess-v10-i1.gif AhMyGoddess-v01.gif AhMyGoddess-v06.gif AhMyGoddess-v10-i2.gif AhMyGoddess-v02-f1.gif AhMyGoddess-v07-f1.gif AhMyGoddess-v10-i3.gif AhMyGoddess-v02.gif AhMyGoddess-v07.gif AhMyGoddess-v10.gif AhMyGoddess-v03-f1.gif AhMyGoddess-v08-f1.gif amg-nt0694_cover.gif AhMyGoddess-v03.gif AhMyGoddess-v08.gif amg-nt0694_i1.gif AhMyGoddess-v04-f1.gif AhMyGoddess-v09-f1.gif amg-nt0694_i2.gif AhMyGoddess-v04.gif AhMyGoddess-v09.gif AhMyGoddess-v05-f1.gif AhMyGoddess-v10-b.gif brezup amg 247> foreach testfile ( *.gif ) foreach -> greyconvert $testfile foreach -> end ppmquant: making histogram... ppmquant: 165 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... ppmtopict: computing colormap... ppmtopict: 4 colors found ppmquant: making histogram... ppmquant: 159 colors found . . . ppmquant: making histogram... ppmquant: 163 colors found ppmquant: choosing 4 colors... ppmquant: mapping image to new colors... ppmtopict: computing colormap... ppmtopict: 4 colors found brezup amg 248> ls AMG_cal-cover.gif AhMyGoddess-v05-f1.pict AhMyGoddess-v10-b.gif AMG_cal-cover.pict AhMyGoddess-v05.gif AhMyGoddess-v10-b.pict AhMyGoddess-v01-f1.gif AhMyGoddess-v05.pict AhMyGoddess-v10-f1.gif AhMyGoddess-v01-f1.pict AhMyGoddess-v06-f1.gif AhMyGoddess-v10-f1.pict AhMyGoddess-v01.gif AhMyGoddess-v06-f1.pict AhMyGoddess-v10-i1.gif . . . AhMyGoddess-v04-f1.gif AhMyGoddess-v08.pict amg-nt0694_i1.gif AhMyGoddess-v04-f1.pict AhMyGoddess-v09-f1.gif amg-nt0694_i1.pict AhMyGoddess-v04.gif AhMyGoddess-v09-f1.pict amg-nt0694_i2.gif AhMyGoddess-v04.pict AhMyGoddess-v09.gif amg-nt0694_i2.pict AhMyGoddess-v05-f1.gif AhMyGoddess-v09.pict
In this example, the foreach testfile ( *.gif ) line could have been replaced with the following variants to achieve identical results:
foreach testfile ( `ls *.gif` ) foreach testfile ( AMG_cal-cover.gif AhMyGoddess-v01-f1.gif ... amg-nt0694_i2.gif )
The while command works similarly, executing its code block while some <condition> holds:
brezup Documents 249> set x = 10 brezup Documents 250> while ( $x > 0 ) while -> echo $x while -> @ x = ( $x - 1 ) while -> end 10 9 8 7 6 5 4 3 2 1 brezup Documents 251> echo $x 0
This example obviously doesn't have much day-to-day applicability. Most while expressions that are actually useful do things like watch for particular events to occur, such as the existence of temporary files or disk space usage of more or less than some value. None of these is particularly easy to demonstrate in a text-only format such as a book, but we expect that you'll get the idea fairly quickly. Table 15.11 shows the conditional operators that can be used to construct the <condition> part of while loops and if conditional statements.
Another common use for the while command is to create infinite loops in the shell. This is a way to do things such as cause a shell command to execute over and over, potentially creating something like a "drop directory" that automatically processes files that are copied to it. For example, if we want to create a directory into which we could copy GIF files, and any file copied into it would have the greyconvert process run on it automatically, and then the GIF files would be deleted, we might try something like the following:
brezup Documents 252> while (1) while -> foreach testfile (*.gif) while -> greyconvert $testfile while -> rm $testfile while -> end while -> sleep 60 while -> end
This while command attempts to loop perpetually (the value of 1 is true for the purposes of a comparison expression) and to internally execute our previous foreach loop to convert any files that match the *.gif pattern in the current directory. An rm command has been added to the foreach loop to remove the GIF file after it has been converted. The sleep 60 command after the foreach command's end causes the while loop to pause for 60 seconds before going on to its end statement and relooping to the top of the while.
Unfortunately, from the command line, this does not quite work properly in all cases; when there are no files that match *.gif, the foreach line fails without creating its loop, and the end on line 4 is mistaken as intended to end the while loop.
Thankfully, even in tcsh you can work around this problem by applying the final topic in our discussion of shell scripts.
Storing Your Automation in Files: Proper Scripts
With all the power available to you directly at the command line, the move to putting shell scripts in files should seem almost anticlimactic in its lack of complexity. As mentioned at the beginning of this chapter, anything that you can type on the command line, you can put in a file, and the system will quite happily execute it for you if you make the file executable. It really is that simple. There is really little more to say, except that putting your script in a file allows you to conveniently separate parts of the execution into separate shells, preventing conflicts such as those just demonstrated earlier with the while and foreach loops.
Any script that is put in a file and directly executed (rather than sourced in your current shell) creates its own shell in which to execute. To use this to make the previous example function properly, we can put the foreach section of the command into its own file:
brezup Documents 253> cat > greyconv.csh #!/bin/csh foreach testfile (*.gif) greyconvert $testfile rm $testfile end brezup Documents 254> chmod 755 greyconv.csh brezup Documents 255> ls -l greyconv.csh -rwxr-xr-x 1 ray staff 75 Jun 23 01:58 greyconv.csh brezup Documents 256> cat greyconv.csh #!/bin/csh foreach testfile (*.gif) greyconvert $testfile rm $testfile end
Then our perpetual while command can be run as
brezup Documents 257> while (1) while -> ./greyconv.csh while -> sleep 60 while -> end
This command will loop perpetually in the current directory, executing the greyconv.csh shell script every 60 seconds. Any file with a .gif extension that is placed in the directory will be discovered and passed through the greyconvert alias that we created earlier, and then the original file will be deleted. This will run perpetually in the directory, enabling any file dropped in to be converted (within 60 seconds) automatically. Because the end of the foreach is in a completely separate shell, it can't accidentally end the while, and everything will work as expected. Of course, if there were a reason to do this on a regular basis, you could put that while loop into its own shell script file. It could be stored and executed as a single command from the command line just like any other command.
Using Shell Parameters and Conditional Execution in the Shell
Two final things to note now that we've introduced independent shells invoked as the result of placing scripts in files: the $argv...$argv[n] list and the corresponding $1...$n command-line argument variables. (Refer to the table of shell variables and the table of alternative variable addressing methods for a refresher on these.) These parameters that can be passed into a shell script by placing values on the command line after the script name. They also provide a good mechanism for us to introduce the if shell command, allowing conditional execution of code blocks.
If we'd like to make the greyconv.csh script somewhat more general, we can make use of the ability to pass arguments into the script. For example, it would be nice to be able to use greyconv.csh on the GIF files in a directory without actually having to be in the directory when we run the while loop. Simultaneously, we might like to have the ability to convert all the files in the current directory without needing to specify a directory name. There is no need to write two different scripts to do this. One script can perform both actions by using conditional statements and checking to see whether a parameter has been passed on the command line. If there is a shell parameter, it can be taken as the directory name in which to work; if there is no shell parameter, the script can operate in the current working directory. The modified version of greyconv.csh is as follows:
#!/bin/csh if ( $?1 == 1 ) then cd $1 endif foreach testfile (*.gif) greyconvert $testfile rm $testfile end
This version of greyconv.csh demonstrates both an if conditional expression and the use of a command-line argument. The if statement checks whether the variable $1 is set (using the $?<variablename> alternative variable addressing to check for existence). If it is set, the script assumes that the value in $1 is the name of a directory, and it cds into that directory before executing the foreach loop to convert and delete the GIF files. greyconv.csh can now be called with a directory, to operate in that directory, or without a directory specified, to operate in the current directory.
We hope that gives you a few ideas for how powerful shell scripts can be, and how you can use them to make your use of Mac OS X much more productive. We've only just scratched the surface in this chapter, and have trivialized some explanations to their simplest case to avoid a chapter that takes half the book. What's here can get you quite a way into shell scripting, and many Unix users with years of experience don't use more than a fraction of what we've covered. Still, if you're looking for more power and more capabilities, don't hesitate to go to the man pages and shell programming specific reference books.
|< Day Day Up >|