Resource Scripts

All of the scripts under Heartbeat's control are called resource scripts. These scripts may include the ability to add or remove a secondary IP address or IP alias, or they may include packet-handling rules (see Chapter 2) in addition to being able to start and stop a service. Heartbeat looks for resource scripts in the Linux Standards Base (LSB) standard init directory (/etc/init.d) and in the Heartbeat /etc/ha.d/resource.d directory.

Heartbeat should always be able to start and stop your resource by running your resource script and passing it the start or stop argument. As illustrated in the iptakeover script, this can be accomplished with a simple case statement like the following:

 #!/bin/bash case $1 in start)     commands to start my resource     ;; stop)     commands to stop my resource     ;; status)     commands to test if I own the resource     ;; *)     echo "Syntax incorrect. You need one of {start|stop|status}"     ;; esac 

The first line, #!/bin/bash, makes this file a bash (Bourne Again Shell) script. The second line tests the first argument passed to the script and attempts to match this argument against one of the lines in the script containing a closing parenthesis. The last match, *), is a wildcard match that says, "If you get this far, match on anything passed as an argument." So if the program flow ends up executing these commands, the script needs to complain that it did not receive a start, stop, or status argument.

Status of the Resource

Heartbeat should always know which computer owns the resource or resources being offered. When you write a script to start or stop a resource, it is important to write the script in such a way that it will accurately determine whether the service is currently being offered by the system. If it is, the script should respond with the word OK, Running, or running when passed the status argument—Heartbeat requires one of these three responses if the service is running. What the script returns when the service is not running doesn't matter to Heartbeat—you can use DOWN or STOPPED, for example. However, do not say not running or not OK for the stopped status. (See the "Using init Scripts as Heartbeat Resource Scripts" section later in this chapter for more information on proposed standard exit codes for Linux.)

Note 

If you want to write temporary scratch files with your resource script and have Heartbeat always remove these files when it first starts up, write your files into the directory /var/lib/heartbeat/rsctmp (on a standard Heartbeat installation on Linux).

Resource Ownership

So how do you write a script to properly determine if the machine it is running on currently owns the resource? You will need to figure out how to do this for your situation, but the following are two sample tests you might perform in your resource script.

Testing for Resource Ownership—Is the Daemon Running?

Red Hat Linux and SuSE Linux both ship with a program called pidof that can test to see if a daemon is running. The pidof program finds the process identifier (PID) of a running daemon when you give it the name of the program file that was used to start the daemon.

When you run the pidof script from within your resource script you should always pass it the full pathname of the daemon you want to search for in the process table. For example, to determine the PID of the sendmail program, you would use this command:

 #pidof /usr/sbin/sendmail 

If sendmail is running, its PID number will be printed, and pidof exits with a return code of zero. If sendmail is not running, pidof does not print anything, and it exits with a nonzero return code.

Note 

See the /etc/init.d/functions script distributed with Red Hat Linux for an example use of the pidof program.

A sample bash shell script that uses pidof to determine whether the sendmail daemon is running (when given the status argument) might look like this:

 #!/bin/bash # case $1 in status)     if /sbin/pidof /usr/sbin/sendmail >/dev/null; then         echo "OK"     else         echo "DOWN"     fi ;; esac 

If you place these commands in a file, such as /etc/ha.d/resource.d/ myresource, and chmod the file to 755, you can then run the script with this command:

 #/etc/ha.d/resource.d/myresource status 

If sendmail is running, the script will return the word OK; if not, the script will say DOWN. (You can stop and start the sendmail daemon for testing purposes by running the script /etc/init.d/myresource and passing it the start or stop argument.)

Testing for Resource Ownership—Do You Own the IP Address?

If you use Heartbeat's built-in ability to automatically failover secondary IP addresses or IP aliases, you do not need to worry about this second test (see the discussion of the IPaddr2 resource script and the discussion of resource groups in Chapter 8). However, if you are using the iptakeover script provided on the CD-ROM, or if you are building a custom configuration that makes changes to network interfaces or the routing table, you may want to check to see if the system really does have the secondary IP address currently configured as part of your script's status test.

To perform this check, the script needs to look at the current secondary IP addresses configured on the system. We can modify the previous script and add the ip command to show the secondary IP address for the eth0 NIC as follows:

 #!/bin/bash # case $1 in status)         if /sbin/pidof /usr/sbin/sendmail >/dev/null; then         ipaliasup=`/sbin/ip addr show dev eth0 | grep 200.100.100.3`         if [ "$ipaliasup" != "" ]; then         echo "OK"         exit 0         fi         fi         echo "DOWN" ;; esac 

Note that this script will only work if you use secondary IP addresses. To look for an ip alias, modify this line in the script:

 ipaliasup=`/sbin/ip addr show dev eth0 | grep 200.100.100.3` 

to match the following:

 ipaliasup=`/sbin/ifconfig eth0 | grep 200.100.100.3` 

This changed line of code will now check to see if the IP alias 200.100.100.3 is associated with the interface eth0. If so, this script assumes the interface is up and working properly, and it returns a status of OK and exits from the script with a return value of 0.

Note 

Testing for a secondary IP address or an IP alias should only be added to resource scripts in special cases; see Chapter 8 for details.

Testing a Resource Script

You can test to make sure the Heartbeat system will work properly with your newly created resource script by running the following command:

 #/usr/lib/heartbeat/ResourceManager status <resource-name> 

where <resource-name> is the name of your newly created resource script in the /etc/rc.d/init.d or /etc/ha.d/resource.d directory. Then tell your shell to display the value of your resource script's return code with this command:

 #echo $? 

The $? must be the next command you type after running the script— otherwise the return value stored in the shell variable ? will be overwritten.

If the return code is 3, it means your script returned something other than OK, Running, or running, which means that Heartbeat thinks that it does not own the resource. If this command returns 0, Heartbeat considers the resource active.

Once you have built a proper resource script and placed it in the /etc/ rc.d/init.d directory or the /etc/ha.d/resource.d directory and thoroughly tested its ability to handle the start, stop, and status arguments, you are ready to configure Heartbeat.

Using init Scripts as Heartbeat Resource Scripts

Most of the /etc/init.d scripts (the scripts used at system boot time by the init process) will already properly handle the start, stop, and status arguments. In fact, the Linux Standard Base Specification (see http://www.linuxbase.org) states that in future releases of Linux, all Linux init scripts running on all versions of Linux that support LSB must implement the following: start, stop, restart, reload, force-reload, and status.

The LSB project also states that if the status argument is sent to an init script it should return 0 if the program is running.

For the moment, however, you need only rely on the convention that a status command should return the word OK, or Running, or running to standard output (or "standard out").^[13] For example, Red Hat Linux scripts, when passed the status argument, normally return a line such as the following example from the /etc/rc.d/init.d/sendmail status command:

 sendmail (pid 4511) is running... 

Heartbeat looks for the word running in this output, and it will ignore everything else, so you do not need to modify Red Hat's init scripts to use them with Heartbeat (unless you want to also add the test for ownership of the secondary IP address or IP alias).

However, once you tell Heartbeat to manage a resource or script, you need to make sure the script does not run during the normal boot (init) process. You can do this by entering this command:

 #chkconfig --del <scriptname> 

where <scriptname> is the name of the file in the /etc/init.d directory. (See Chapter 1 for more information about the chkconfig command.)

The init script must not run as part of the normal boot (init) process, because you want Heartbeat to decide when to start (and stop) the daemon. If you start the Heartbeat program but already have a daemon running that you have asked Heartbeat to control, you will see a message like the following in the /var/log/messages file:

 WARNING: Non-idle resources will affect resource takeback. 

If you see this error message when you start Heartbeat, you should stop all of the resources you have asked Heartbeat to manage, remove them from the init process (with the chkconfig --del <scriptname> command), and then restart the Heartbeat daemon (or reboot) to put things in a proper state.

^[13]Programs and shell scripts can return output to standard output or to standard error. This allows programmers to control, through the use of redirection symbols, where the message will end up—see the "REDIRECTION" section of the bash man page for details.