14.3 Expect: Automating Interactive Programs

Don Libes describes his Expect package as "a software suite for automating interactive tools." Expect lets you drive interactive programs from a script. The shell lets you do that too, but only to a very limited extent and not in any general way. Expect lets a script feed input to commands and programs that demand their input from the terminal meaning /dev/tty. It also allows different things to happen depending on the output it gets back, which goes far beyond what the shell offers. If this doesn't sound like any big deal and it didn't to me, at first read on and consider some of the examples in this section. Expect is actually quite addictive once you begin to figure out what it's good for.

For more information on Expect, see its home page at http://expect.nist.gov. The book Exploring Expect, by Don Libes (O'Reilly & Associates) is also very helpful.

Conceptually, Expect is a chat script^[7] generalized to the entire Unix universe. Structurally, Expect is actually an extension to another programming language called Tcl. This means that Expect adds commands and functionality to the Tcl language. It also means that to build and use Expect, you must also obtain and build Tcl.

^[7] Traditionally, a chat script defines the login conversation that occurs between two computers when they connect, and it is made up of a series of expect-send pairs, as in this example:

14.3.1 A First Example: Testing User Environments

The following Expect script illustrates many of the facility's basic features. It is used to run the /usr/local/sbin/test_user script from a user's account. This shell script tests various security-related features of the user's runtime environment, and it needs to be run as the relevant user. This Expect script allows it to be run by the system administrator:

#!/usr/local/bin/expect                  Executable location may vary. # run_test_user - check security of user acct set user [lindex $argv 0]                # set user to first argument  spawn /bin/sh                            # start a conversation  expect "#"  send "su - $user\r"  expect -re "..* $"  send "/usr/local/sbin/test_user >> /tmp/results\r"  expect -re "..* $"  send "exit\r"  expect "#"  close                                    # end the conversation

The first command stores the username specified as the Expect script's argument in the variable user. Arguments are stored automatically in the array argv, and the lindex Tcl function extracts the first element from the array (numbering begins at zero). In Tcl, square brackets are used to evaluate a function or command and use its return value in another command.

The spawn command begins a conversation; the command specified as its argument is started in a subshell in this case, the command itself is just a shell and the Expect script interacts with it via expect and send commands.

expect commands search the output of the spawned command for the first match of a pattern or a regular expression (the latter is indicated by its -re option). When a match is found, the script goes on to the next command; put another way, the script blocks until the desired string is encountered.

send commands provide input to a spawned process (enclosed in quotation marks and usually ending with \r, indicating a carriage return). send commands can include dereferenced variables (as in the first one in the preceding script, whose string contains $user).

Thus, the first expect command waits for a sharp sign (#) to appear (the root prompt, since the script will be run by root). The following send command transmits a command like su - chavez to the spawned shell. Similarly, the next expect command waits for at least one character and then the end of the output (the latter is denoted by the dollar sign), and the following send command runs the script. Once the next prompt is received, the script sends an exit command to the shell created by the su command; when the root prompt reappears, indicating that the sub-subshell has exited, the script executes the close command, which terminates the spawned command.

This sort of staged conversation represents the simplest use of Expect, although this script also illustrates that Expect can allow you to automate activities that are possible in no other way. Once an Expect script exists, it can be called from a normal shell script just like any other command. For example, this C shell script could be used to automate the testing of a group of user accounts:

#!/bin/csh # test_em_all - security-check user accounts unset path; setenv PATH "/usr/bin:/bin" foreach u (`cat /usr/local/admin/check_users`)     /usr/local/sbin/run_test_user $u  end

14.3.2 A Timed Prompt

Here is a timed prompt function. It displays a prompt and waits for user input. If no input is received within a set period of time, the function returns some default value:

#!/usr/local/bin/expect # timed_prompt - prompt with timeout # args: [prompt [default [timeout]]] # process arguments set prompt [lindex $argv 0]  set response [lindex $argv 1]  set tout [lindex $argv 2]  if {"$prompt" == ""} {set prompt "Enter response"}  if {"$tout" == ""} {set tout 5} set clean_up 1  send_tty "$prompt: "  set timeout $tout  expect "\n" {     set response [string trimright "$expect_out(buffer)" "\n"]     set clean_up 0     }  if {$clean_up == 1} {send_tty "\n"}  send "$response"

The first section of the script processes it arguments, assigning them to variables and applying default values. This section of the script is pure Tcl, and it illustrates the language's if statement everything goes in curly braces. Both these if statements have only a single command in their body, but we'll see more complex examples a bit later. The second if statement also illustrates one of the nicest features of Tcl: the complete equivalence of integers and strings.

The second part of the script does the actual prompting. The send_tty command displays a string on the screen (regardless of any other current conversations), in this case, the prompt string. The set timeout command specifies a timeout period for subsequent expect commands (in seconds, with -1 indicating no timeout).

Given that Expect has built-in timeouts, all the expect command has to look for is a newline (indicating that the user has pressed the return key, ending her input). If a newline is found before the timeout period expires, the response variable is assigned the value that the user typed in, minus the final carriage return (via the string Tcl function); if not, then response retains its previous value (the default value specified as the script's second argument). The final command transmits the response (or default value) to the calling script.

The clean_up variable is used to keep track of whether a response was entered; if not, a newline is sent to the screen after the expect command times out to avoid a Unix prompt running into the lingering script prompt in an ugly way.

Here is how timed_prompt might be used within a shell script:

ishell=`timed_prompt "Enter desired shell [/bin/sh]" "/bin/sh" 10`

14.3.3 Repeating a Command Over and Over

In this section, we'll look at another task that is possible in a shell program but is much easier in Expect. This script, loop, runs a command continuously until any character is entered at the keyboard (in a shell script version, you'd have to use CTRL-C to exit). Such a command is very useful for real-time monitoring of any system phenomenon (general system performance, watching a particular process, following some security-related event, and so on).

Here is the script, loop:

#!/usr/local/bin/expect # loop - repeat command until a key is pressed # args: command [timeout] set cmd [lindex $argv 0]  set timeo [lindex $argv 1]  if {"$cmd" == ""} {     send "Usage: loop <command> \[interval]\n"     exit     }  if {"$timeo" == ""} {set timeo 3} set timeout $timeo  set done ""  while {"$done" == ""} {     system /usr/bin/clear                # run the Unix clear commmand     system /usr/bin/$cmd                 # run the specified command     stty raw                             # put terminal in raw mode     expect "?" {                         # wait for a character        set done 1        }     stty -raw                            # restore terminal to normal mode     }  exit

The first section of the script again processes command-line arguments and sets the timeout period to the default value of three seconds if necessary. In this case, this means that the desired command will be run once every three seconds.

The second section of the script uses a while loop to run the command; as with the if command, the condition and body of the loop are both enclosed in curly braces. The first two lines within the loop use the system command to run a Unix command without starting a conversation (in contrast to spawn), in this case, the clear command, followed by the desired command. The latter command is assumed to be in /usr/bin, but you could modify the script to allow any command to be run. However, should you choose to do so, make sure that a full pathname is included for the command.

The stty raw command puts the terminal in raw mode, so that the subsequent expect command will be able to match any single character. When a match is found, the variable done is assigned the value 1, which causes the while loop to terminate after the terminal has been restored to its normal mode (contrast this with placing the exit command in the body of the expect statement).

14.3.4 Automating Configuration File Distribution

The script we'll look at in this section will illustrate Expect's ability to take different actions depending on what is "said" in the course of a conversation. This script distributes the /etc/hosts and /etc/shosts.equiv files to the systems specified as the script's command line arguments.^[8]

^[8] This script executes in an isolated and trusted network. The actions it performs may or may not make sense in your environment. However, the Expect concepts will still be useful to you. Note that you could also use the rdist facility to perform this function.

Here is the first part of the script, which obtains the root password:

#!/usr/local/bin/expect # hostdist - distribute hosts and shosts.equiv files set timeout -1 # get the root password (once!)  stty -echo                               # turn off echoing  send_user "# "                           # prompt for password  expect_user -re "(.*)\n"                 # get and remember it  # assign password to variable send_user "\n"  set passwd $expect_out(1,string)  stty echo                                # turn echo back on

The first command turns off expect timeouts, and then the stty command disables echoing while the root password is entered. The expect command, which consumes the entered password, places parentheses around part of the regular expression. These have no effect on whether it is matched in this case, but it does allow whatever matches the enclosed portion of the regular expression to be accessed as a unit later on. This is done two lines later, in the set command, which assigns the saved password to the variable passwd (expect_out is an array that contains the results from the most recent expect command). Once the root password has been obtained, echoing is turned back on.

The next section of the script sets up a loop over the hosts to be updated:

set num [llength $argv]                     # number of hosts  incr num -1                                 # account for 0-based counting  for {set index 0} {$index <= $num} {incr index} {     set host [lindex $argv $index]     spawn /usr/bin/ssh $host     expect {        -re "(timed out)|(timeout)" {         # ssh failed          continue                           # just go on to next host          }       -re ".*> *$" {}                       # got a prompt       }

The llength Tcl function returns the length of a list in this case, this is equivalent to the number of elements in the array argv, and the incr command adds a number to a variable (by default, 1), so after the first two commands, the variable num holds one less than the number of hosts specified on the command line.

The for loop that begins on the next line makes up the better part of the hostdist script. A Tcl for command has the following general form:

for {initialize} {condition} {update} {    body of the loop  }

(Its structure is very like the C for loop.) The initialize clause holds commands to be run before the loop's first iteration, and it usually serves to initialize the loop variable (as it does in our example). The condition clause contains a test that determines whether the loop should continue with the next iteration; the update clause is run after each loop iteration (and before the next test of the condition) and is used to increment the loop variable index in our loop.

The first few commands within the body of the loop assign the next hostname in argv to the variable host and then execute a spawn command running rlogin to that host. The subsequent expect command is a bit more complex than those we've seen before in that it contains two patterns rather than just one (all enclosed within a pair of curly braces). The first pattern looks for a TCP/IP timeout error message, which would indicate that the ssh command failed; in this case, the continue command causes the script to jump immediately to the next loop iteration. The second pattern searches for a prompt string, which is assumed to end with a greater-than sign (as is true on my systems).

When multiple patterns are included within a single expect command in this way, the first one that matches is used. If more than one pattern matches, the one that occurs earliest in the list is used.

The next section of the script copies the two files from system iago to /tmp on the current remote host. These commands are executed as the user running hostdist because the systems don't trust remote root users:

   # copy the files     send "/usr/bin/rcp iago:/etc/hosts /tmp\r"     expect -re ".*> *$"                            # wait for prompt     send "/usr/bin/rcp iago:/etc/shosts.equiv /tmp\r"     expect -re ".*> *$"                            # wait for prompt     send "/bin/su\r"     expect "assword:"     send "$passwd\r"

Once the two rcp commands complete, an su command is given, and the saved root password is sent in response to the password prompt.

The next expect command handles commands that are executed as root:

   expect {        -re "# $" {                                 # got a root prompt          # install new files          send "/usr/bin/cp /tmp/hosts /etc/hosts\r"          expect "# $"          send "/usr/bin/cp /tmp/shosts.equiv /etc/shosts.equiv\r"          expect "# $"          send "/usr/bin/chmod 644 /etc/shosts.equiv\r"          expect "# $"          send "/usr/bin/rm -f /tmp/hosts /tmp/shosts.equiv\r"          expect "# $"          send "exit\r"                            # exit su shell          expect ".*> *$"                          # wait for prompt          }        -re ".*> *$" {}                             # regular prompt: su failed       }

The first pattern will match a normal root prompt. When this is received, the script runs the commands that copy the files in /tmp to /etc, set their permissions correctly, and remove the originals from /tmp are executed. Afterwards, the script uses an exit command to end the su shell.

The second pattern in the first expect command matches the normal shell prompt. If it is matched, the su command failed for some reason, and no action is taken (indicated by the empty curly braces following the pattern).

Here is the remainder of the script:

   send "logout\r"    # all done with this host     expect "?"         # accept anything     close              # terminate spawn command     }                  # end for loop  exit

Once the su shell has terminated (if it ever started), the script sends a logout command to the ssh shell, waits for its output, and terminates the current conversation via the close command.

14.3.5 Keep Trying Until It Works

As our final Expect example, we'll consider a script that repeatedly attempts an operation until it succeeds a canonical use of Expect. In this case, the operation is to repeatedly dial an electronic mail forwarding service until a successful connection is made.

Here is the script, pester:

#!/usr/local/bin/expect # pester - keep calling until we get through set done 0                         # did we get through yet?  for {set index 1} {$index <= 2000} {incr index} {     system "call-command"           # call ISP     while {$done == 0} {            # continuously check status       spawn /usr/local/admin/isp_stat       expect {                     # branch depending on results          -re "(SENDING)|(RECEIVING)" {             set done 1             # success, so set done to 1             }          -re "NO DEV" {             sleep 120              # line in use; wait a bit             break             }          -re "FAILED" {             break                  # poll failed so try again             }          }                         # end expect        }                            # end while     if {$done == 1} {break}      # if we succeeded, end the for loop     }                               # end for  exit

This script actually calls the ISP site only 2000 times before giving up, which is a bit of a hack, but it offers another example of a for loop. The system command executes the appropriate command to initiate the connection, and the subsequent while loop runs a status script which provides a snapshot of current activity continuously until a connection is established. The expect command contains three slightly complex regular expressions designed to match the different possible output that the status script can produce (it functions similarly to a case construct).

The break command breaks out of the innermost construct in which it is embedded (i.e., currently in effect). Thus, the break commands in the expect command bodies jump out of the while loop, while the final break command (making up the if body) ends the for loop.