Understanding Shell Scripts
Have you ever had a task that you needed to do over and over that took a lot of typing on the command line? Do you ever think to yourself, “Wow, I wish there was just one command I could type to do all this of this”? Maybe a shell script is what you’re after.
Shell scripts are the equivalent of batch files in MS-DOS, and can contain long lists of commands, complex flow control, arithmetic evaluations, user-defined variables, user-defined functions, and sophisticated condition testing. Shell scripts are capable of handling everything from simple one-line commands to something as complex as starting up your Fedora system.
In fact, as you will read in this chapter, Fedora does just that. It uses shell scripts (/etc/rc.d/rc.sysinit and /etc/rc) to check and mount all your filesystems, set up your consoles, configure your network, launch all your system services, and eventually provide you with your login screen. While there are nearly a dozen different shells available in Fedora, the default shell is called bash, the Bourne-Again SHell.
Executing and debugging shell scripts
One of the primary advantages of shell scripts is that they can be opened in any text editor to see what they do. A big disadvantage is that shell scripts often execute more slowly than compiled programs. There are two basic ways to execute a shell script:
-
The filename is used as an argument to the shell ( as in bash myscript). In this method, the file does not need to be executable; it just contains a list of shell commands. The shell specified on the command line is used to interpret the commands in the script file. This is most common for quick, simple tasks.
-
The shell script may also have the name of the interpreter placed in the first line of the script preceeded by #! (as in #!/bin/bash), and have its execute bit set (using chmod +x). You can then run your script just like any other program in your path simply by typing the name of the script on the command line.
Cross-reference: See Chapter 4 for more details on chmod and read/write/execute permissions.
When scripts are executed in either manner, options to the program may be specified on the command line. Anything following the name of the script is referred to as a command-line argument.
As with writing any software, there is no substitute to clear and thoughtful design and lots of comments. The pound sign (#) prefaces comments and can take up an entire line or exist on
the same line as script code. It’s best to implement more complex shell scripts in stages, making sure the logic is sound at each step before continuing. Here are a few good, concise tips to make sure things are working as expected during testing:
-
Place an echo statement at the beginning of lines within the body of a loop. That way, rather than executing the code, you can see what will be executed without making any permanent changes.
-
To achieve the same goal, you could place dummy echo statements throughout the code. If these lines get printed, you know the correct logic branch is being taken.
-
You could use set -x near the beginning of the script to display each command that is executed or launch your scripts using bash –x myscript.
Understanding shell variables
Often within a shell script, you want to reuse certain items of information. During the course of processing the shell script, the name or number representing this information may change. To store information used by a shell script in such a way that it can be easily reused, you can set variables. Variable names within shell scripts are case-sensitive and can be defined in the following manner:
NAME=value
The first part of a variable is the variable name, and the second part is the value set for that name. Variables can be assigned from constants, like text or numbers. This is useful for initializing values or saving lots of typing for long constants. Here are examples where variables are set to a string of characters (CITY) and a numeric value (PI):
CITY="Springfield" PI=3.14159265
Variables can contain the output of a command or command sequence. You can accomplish this by enclosing the command in backticks (`).
This is a great way to get information that can change from computer to computer or from day to day. Here we set the output of the uname -n command to the MACHINE variable. Then we use parentheses to set NUM_FILES to the number of files in the current directory by piping (|) the output of the ls command to the word count command (wc -l).
MACHINE=`uname –n` NUM_FILES=`/bin/ls | wc –l`
Variables can also contain the value of other variables. This is useful when you have to preserve a value that will change so you can use it later in the script. Here BALANCE is set to the value of the CurBalance variable.
BALANCE="$CurBalance"
| Note | : When assigning variables, use only the variable name (for example, BALANCE). When referenced, meaning you want the value of the variable, precede it with a dollar sign (as in $CurBalance). |
Special Shell Variables
There are special variables that the shell assigns for you . The most commonly used variables are called the positional parameters or command line arguments and are referenced as $0, $1, $2, $3…$n. $0 is special and is assigned the name used to invoke your script; the remainder are assigned the values of the parameters passed on the command line. For instance, if the shell script named myscript were called as:
myscript foo bar
the positional parameter $0 would be myscript, $1 would be foo, and $2 would be bar.
Another variable, $#, tells you how many parameters your script was given. In our example, $# would be 2. Another particularly useful special shell variable is $?, which receives the exit status of the last command executed. Typically, a value of zero means everything is okay, and anything other than zero indicates an error of some kind. For a complete list of special shell variables, refer to the bash man page .
Parameter expansion in bash
As mentioned earlier, if you want the value of a variable, you precede it with a $ (for example, $CITY). This is really just a shorthand for the notation ${CITY}; curly braces are used when the value of the parameter needs to be placed next to other text without a space. Bash has special rules that allow you to expand the value of a variable in different ways. Going into all the rules is probably a little overkill for a quick introduction to shell scripts, but Table 12-1 presents some common constructs that you’re likely to see in bash scripts you find on your Fedora box.
| Construction | Meaning |
|---|---|
| ${var:-value} | If variable is unset or empty, expand this to value |
| ${var#pattern} | Chop the shortest match for pattern from the front of var’s value |
| ${var##pattern} | Chop the longest match for pattern from the front of var’s value |
| ${var%pattern} | Chop the shortest match for pattern from the end of var’s value |
| ${var%%pattern} | Chop the longest match for pattern from the end of var’s value |
Try typing the following commands from a shell to test out how parameter expansion works:
$ THIS="Example" $ THIS=${THIS:- "Not Set"} $ THAT=${THAT:- "Not Set"} $ echo $THIS Example $ echo $THAT Not Set In the examples here, the THIS variable is initially set to the word Example. In the next two lines, the THIS and THAT variables are set to their current values or to Not Set, if they are not currently set. Notice that because we just set THIS to the string Example, when we echo the value of THIS it appears as Example. However, since THAT was not set, it appears as Not Set.
| Note | For the rest of this section, I show how variables and commands may appear in a shell script. To try out any of those examples, however, you can simply type them into a shell as shown in the previous example. |
In the following example, set MYFILENAME to /home/digby/myfile.txt. Next, the FILE variable is set to myfile.txt and DIR is set to /home/digby. In the NAME variable, the file name is cut down to simply myfile, then in the EXTENSION variable the file extension is set to txt. (To try these out, you can type them at a shell prompt as we did with the previous example, then echo the value of each variable to see how it is set.)
MYFILENAME="/home/digby/myfile.txt" FILE=${MYFILENAME##*/} #FILE becomes "myfile.txt" DIR=${MYFILENAME%/*} #DIR becomes "/home/digby" NAME=${FILE%.*} #NAME becomes "myfile" EXTENSION=${FILE#*.} #EXTENSION becomes "txt"
Performing arithmetic in shell scripts
Bash used untyped variables, meaning it normally treats variables as strings or text, but can change them on the fly if you want it to. Unless you tell it otherwise with declare, your variables are just a bunch of letters to bash. But when you start trying to do arithmetic with them, bash will convert them to integers if it can. This makes it possible to do some fairly complex arithmetic in bash.
Integer arithmetic can be performed using the built-in let command or through the external expr or bc commands. After setting the variable BIGNUM value to 1024, the three commands that follow would all store the value 64 in the RESULT variable:
BIGNUM=1024 let RESULT=$BIGNUM/16 RESULT=`expr $BIGNUM / 16` RESULT=`echo "$BIGNUM / 16" | bc –l`
| Note | While most elements of shell scripts are relatively freeform (where whitespace, such as spaces or tabs, is insignificant), both let and expr are particular about spacing. The let command insists on no spaces between each operand and the mathematical operator, whereas the syntax of the expr command requires whitespace between each operand and its operator. In contrast to those, bc isn’t picky about spaces, but can be trickier to use because it does floating-point arithmetic. |
To see a complete list of the kinds of arithmetic you can perform using the let command, type help let at the bash prompt.
Using programming constructs in shell scripts
One of the features that make shell scripts so powerful is that their implementation of looping and conditional execution constructs is similar to those found in more complex scripting and programming languages. You can use several different types of loops, depending on your needs.
The “if… then” statements
The most commonly used programming construct is conditional execution, or the “if” statement. It is used to perform actions only under certain conditions. There are several variations, depending on whether you’re testing one thing, or want to do one thing if a condition is true, but another thing if a condition is false, or if you want to test several things one after the other.
The first if…then example tests if VARIABLE is set to the number 1. If it is, then the echo command is used to say that it is set to 1.
if [ $VARIABLE -eq 1 ] ; then echo "The variable is 1" fi
Instead of using –eq, you can use the equals sign (=), as shown in the following example. Using the else statement, different words can be echoed if the criterion of the if statement isn't met ($STRING = "Friday").
if [ $STRING = "Friday" ] ; then echo "WhooHoo! Friday!" else echo "Will Friday ever get here?" fi
You can also reverse tests with an exclamation mark (!). In the following example, if STRING is not Monday, then "At least it's not Monday" is echoed.
if [ $STRING != "Monday" ] ; then echo "At least it’s not Monday" fi
In the following example, elif is used to test for an additional condition (is filename a file or a directory).
if [ -f $filename ] ; then echo "$filename is a regular file" elif [ -d $filename ] ; then echo "$filename is a directory" else echo "I have no idea what $filename is" fi
As you can see from the preceding examples, the condition you are testing is placed between square brackets [ ]. When a test expression is evaluated, it will return either a value of 0, meaning that it is true, or a 1, meaning that it is false. Table 12-2 lists the conditions that are testable and is quite a handy reference. (If you’re in a hurry, you can type help test on the command line to get the same information.)
| Operator | What Is Being Tested? |
|---|---|
| -a file | Does the file exist? (same as –e) |
| -b file | Is the file a special block device? |
| -c file | Is the file character special (for example, a character device)? |
| -d file | Is the file a directory? |
| -e file | Does the file exist? (same as -a) |
| -f file | Does the file exist, and is it a regular file (for example, not a directory, socket, pipe, link, or device file)? |
| -g file | Does the file have the set-group-id bit set? |
| -h file | Is the file a symbolic link? (same as –L) |
| -k file | Does the file have the sticky bit set? |
| -L file | Is the file a symbolic link? |
| -n string | Is the length of the string greater than 0 bytes? |
| -O file | Do you own the file? |
| -p file | Is the file a named pipe? |
| -r file | Is the file readable by you? |
| -s file | Does the file exist, and is it larger than 0 bytes? |
| -S file | Does the file exist, and is it a socket? |
| -t fd | Is the file descriptor connected to a terminal? |
| -u file | Does the file have the set-user-id bit set? |
| -w file | Is the file writable by you? |
| -x file | Is the file executable by you? |
| -z string | Is the length of the string 0 (zero) bytes? |
| expr1 -a expr2 | Are both the first expression and the second expression true? |
| expr1 -o expr2 | Is either of the two expressions true? |
| file1 -nt file2 | Is the first file newer than the second file (using the modification timestamp)? |
| file1 -ot file2 | Is the first file older than the second file (using the modification timestamp)? |
| file1 -ef file2 | Are the two files associated by a link (a hard link or a symbolic link)? |
| var1 = var2 | Is the first variable equal to the second variable? |
| var1 -eq var2 | Is the first variable equal to the second variable? |
| var1 -ge var2 | Is the first variable greater than or equal to the second variable? |
| var1 -gt var2 | Is the first variable greater than the second variable? |
| var1 -le var2 | Is the first variable less than or equal to the second variable? |
| var1 -lt var2 | Is the first variable less than the second variable? |
| var1 != var2 | Is the first variable not equal to the second variable? |
| var1 -ne var2 | Is the first variable not equal to the second variable? |
There is also a special shorthand method of performing tests that can be useful for simple one- command actions. In the following example, the two pipes (||) indicate that if the directory being tested for doesn’t exist (-d dirname), then make the directory (mkdir $dirname).
# [ test ] || {action} # Perform simple single command {action} if test is false [ -d $dirname ] || mkdir $dirname Instead of pipes, you can use two ampersands to test if something is true. In the following example, a command is being tested to see if it includes at least three command-line arguments.
# [ test ] && {action} # Perform simple single command {action} if test is true [ $# -ge 3 ] && echo "There are at least 3 command line arguments."
The case command
Another frequently used construct is the case command. Similar to a switch statement in programming languages, this can take the place of several nested if statements. A general form of the case statement is as follows:
case "VAR" in Result1) { body };; Result2) { body };; *) { body } ;; esac One use for the case command might be to help with your backups. The following case statement tests for the first three letters of the current day (case `date +%a` in). Then, depending on the day, a particular backup directory (BACKUP) and tape drive (TAPE) is set.
# Our VAR doesn’t have to be a variable, # it can be the output of a command as well # Perform action based on day of week case `date +%a` in "Mon") BACKUP=/home/myproject/data0 TAPE=/dev/rft0 # Note the use of the double semi-colon to end each option ;; # Note the use of the "|" to mean "or" "Tue" | "Thu") BACKUP=/home/myproject/data1 TAPE=/dev/rft1 ;; "Wed" | "Fri") BACKUP=/home/myproject/data2 TAPE=/dev/rft2 ;; # Don’t do backups on the weekend. *) BACKUP="none" TAPE=/dev/null ;; esac
The asterisk (*) is used as a catchall, similar to the default keyword in the C programming language. In this example, if none of the other entries are matched on the way down the loop, the asterisk is matched, and the value of BACKUP becomes none. Note the use of esac, or case spelled backwards, to end the case statement.
The “for . . . do” loop
Loops are used to perform actions over and over again until a condition is met or until all data has been processed. One of the most commonly used loops is the for...do loop. It iterates through a list of values, executing the body of the loop for each element in the list. The syntax and a few examples are presented here:
for VAR in LIST do { body } done The for loop assigns the values in LIST to VAR one at a time. Then for each value, the body in braces between do and done is executed. VAR can be any variable name, and LIST can be composed of pretty much any list of values or anything that generates a list.
for NUMBER in 0 1 2 3 4 5 6 7 8 9 do echo The number is $NUMBER done for FILE in `/bin/ls` do echo $FILE done # You can also write it this way, which is somewhat cleaner. for NAME in John Paul Ringo George ; do echo $NAME is my favorite Beatle done
Each element in the LIST is separated from the next by whitespace. This can cause trouble if you’re not careful, since some commands, like ls –l, output multiple fields per line, each separated by whitespace.
If you’re a die-hard C programmer, bash allows you to use C syntax to control your loops:
LIMIT=10 # Double parentheses, and no $ on LIMIT even though it's a variable! for ((a=1; a <= LIMIT ; a++)) ; do echo "$a" done
The “while . . . do” and “until . . . do” loops
Two other possible looping constructs are the while. . .do loop and the until. . .do loop. The structure of each is presented here:
while condition until condition do do { body } { body } done done The while statement executes while the condition is true. The until statement executes until the condition is true, in other words, while the condition is false.
Here is an example of a while loop that will output the number 0123456789:
N=0 while [ $N –lt 10 ] ; do echo –n $N let N=$N+1 done
Another way to output the number 0123456789 is to use an until loop as follows:
N=0 until [ $N –eq 10 ] ; do echo –n $N let N=$N+1 done
Some useful external programs
Bash is great and has lots of built-in commands, but it usually needs some help to do anything really useful. Some of the most common useful programs you’ll see used are grep, cut, tr, and sed. Like all the best UNIX tools, most of these programs are designed to work with standard input and standard output, so you can easily use them with pipes and shell scripts.
The General Regular Expression Parser (grep)
The name sounds intimidating, but grep is just a way to find patterns in files or text. Think of it as a useful search tool. Getting really good with regular expressions is quite a challenge, but many useful things can be accomplished with just the simplest forms.
For example, you can display a list of all regular user accounts by using grep to search for all lines that contain the text /home in the /etc/passwd file as follows:
grep /home /etc/passwd
Or you could find all environment variables that begin with HO using the following command:
env | grep ^HO
To find a list of options to use with the grep command, type man grep.
Remove sections of lines of text (cut)
The cut command can extract specific fields from a line of text or from files. It is very useful for parsing system configuration files into easy-to-digest chunks. You can specify the field separator you want to use and the fields you want, or you can break a line up based on bytes.
The following example lists all home directories of users on your system. Using an earlier example of the grep command, this line pipes a list of regular users from the /etc/passwd file, then displays the sixth field (-f6) as delimited by a colon (-d':').
grep /home /etc/passwd | cut –f6 –d':'
Translate or delete characters (tr)
The tr command is a character-based translator that can be used to replace one character or set of characters with another or to remove a character from a line of text.
The following example translates all uppercase letters to lowercase letters and displays the words "mixed upper and lower case" as a result:
FOO="Mixed UPpEr aNd LoWeR cAsE" echo $FOO | tr [A-Z] [a-z]
In the next example, the tr command is used on a list of file names to rename any files in that list so that any spaces contained in a file name are translated into underscores:
for file in * ; do d=`echo $file | tr [:blank:] [_]` ["$file" = "-d" ] || mv "$file" "$d" done
The Stream Editor (sed)
The sed command is a simple scriptable editor, and as such can perform only simple edits, such as removing lines that have text matching a certain pattern, replacing one pattern of characters with another, and other simple edits. To get a better idea of how sed scripts work, there’s no substitute for the online documentation, but here are some examples of common uses.
You can use the sed command to essentially do what we did earlier with the grep example: search the /etc/passwd file for the word home. Here the sed command searches the entire /etc/passwd file, searches for the word home, and prints any line containing the word home.
sed –n '/home/p' /etc/passwd
In this example, sed searches the file somefile.txt and replaces every instance of the string Mac with Linux. The output is then sent to the fixed_file.txt file.
sed 's/Mac/Linux/g' somefile.txt > fixed_file.txt
You can get the same result using a pipe:
cat somefile.txt | sed 's/Mac/Linux/g' > fixed_file.txt
By searching for a pattern and replacing it with a null pattern, you delete the original pattern. This example searches the contents of the somefile.txt file and replaces extra blank spaces at the end of each line (s/ *$) with nothing (//). Results go to the fixed_file.txt file.
cat somefile.txt | sed 's/ *$//' > fixed_file.txt
Trying some simple shell scripts
Sometimes the simplest of scripts can be the most useful. If you type the same sequence of commands repetitively, it makes sense to store those commands (once!) in a file. Here are a couple of simple, but useful, shell scripts.
A simple telephone list
This idea has been handed down from generation to generation of old UNIX hacks. It’s really quite simple, but it employs several of the concepts just introduced.
#!/bin/bash # (@)/ph # A very simple telephone list # Type "ph new name number" to add to the list, or # just type "ph name" to get a phone number PHONELIST=~/.phonelist.txt # If no command line parameters ($#), there # is a problem, so ask what they’re talking about. if [ $# -lt 1 ] ; then echo "Whose phone number did you want? " exit 1 fi # Did you want to add a new phone number? if ["$1" = "new" ] ; then shift echo $* >> $PHONELIST echo $* added to database exit 0 fi # Nope. But does the file have anyting in it yet? # This might be our first time using it, after all. if [ ! -s $PHONELIST ] ; then echo "No names in the phone list yet! " exit 1 else grep –q "$*" $PHONELIST # Quietly search the file if [ $? –ne 0 ] ; then # Did we find anything? echo "Sorry, that name was not found in the phone list" exit 1 else grep "$*" $PHONELIST fi fi exit 0
A Simple Backup Script
Since nothing works forever and mistakes happen, backups are just a fact of life when dealing with computer data. This simple script will back up all the data in the home directories of all the users on your Fedora system.
#!/bin/bash # (@)/my_backup # A very simple backup script # TAPE=/dev/rft0 # Rewind the tape device $TAPE mt $TAPE rew # Get a list of home directories HOMES=`grep /home /etc/passwd | cut –f6 –d’: ’` # Backup the data in those directories tar cvf $TAPE $HOMES # Rewind and eject the tape. mt $TAPE rewoffl
| Cross-reference: | See Chapter 13 for details on backing up and restoring files. |
EAN: 2147483647
Pages: 286