Chapter 8: Heartbeat Resources and Maintenance | Linux Enterprise Cluster: Build a Highly Available Cluster with Commodity Hardware and Free Software

This chapter describes how to use the high-availability resources file, haresources, to control resources on a pair of servers^[1] running Heartbeat. We will also explore some of the common maintenance tasks required to keep the Heartbeat high-availability system functioning properly.

The Haresources File Syntax

The /etc/ha.d/haresources file must be the same on both the primary and the backup Heartbeat servers.

Each line in the haresources file usually contains the following:

The name of the server where the resource should normally run (the primary server), followed by a space or tab.
An (optional) IP alias that Heartbeat should add to the system before launching the resource, followed by a space. (The IP alias definition may include a network subnet mask and a broadcast address separated from each other by the forward slash (/) character.)
A resource script (the script used to start and stop the resource) located in either the /etc/init.d or the /etc/ha.d/resource.d directory.^[2] If arguments need to be passed to the resource script, they are added after two colons and are separated from each other by two colons.

Additional resource scripts can be added to the line using the space character as a separator.

Note

If you need to create a haresources line that is longer than the line of text that fits on your screen, you can add the backslash character (\) to indicate that the haresources entry continues on the next line.

A simplified summary of this syntax, for a single line with two resources, each with two arguments, looks like this:

 primary-server [IPaddress] resource1[::arg1::arg2] [resource2[::arg1::arg2]

In practice, this line might look like the following on a server called primary.mydomain.com running sendmail and httpd on IP address 209.100.100.3:

 primary.mydomain.com 209.100.100.3 sendmail httpd

Let's examine each element of the haresources file in more detail.

Haresources File Syntax: Primary-Server Name

The primary-server name you enter at the start of the haresources line should match one of the server names you've already specified in the /etc/ha.d/ha.cf file. It should also match the name returned by the uname -n command on the primary server.

Haresources File Syntax: IP Alias

Although it is not required, an IP alias^[3] is usually specified in the haresources file. This IP alias can then be offered from either the primary or the backup server, depending upon which system is healthy. For example:

 primary.mydomain.com 209.100.100.3

Heartbeat will add 209.100.100.3 as an IP alias to one of the existing NICs connected to the system and send Gratuitous ARP^[4] broadcasts out of this NIC to the locally connected computers when it first starts up. It will only do this on the backup server if the primary server goes down.

Actually, when Heartbeat sees an IP address in the haresources file, it runs the resource script in the /etc/ha.d/resource.d directory called IPaddr and passes it the requested IP address as an argument. The /etc/ha.d/ resource.d/IPaddr script then calls the program included with the Heartbeat package, findif (find interface), and passes it the IP alias you want to add. This program then automatically selects the physical NIC that this IP alias should be added to, based on the kernel's network routing table.^[5] If the findif program cannot locate an interface to add the IP alias to, it will complain in the /var/log/messages file with a message such as the following:

 heartbeat: ERROR: unable to find an interface for 200.100.100.3

Note

You cannot place the primary IP address for the interface in the haresources file. Heartbeat can add an IP alias to an existing interface, but it cannot be used to bring up the primary IP address on an interface.

Heartbeat's Automated Network Interface Card Selection Process

Heartbeat uses the findif program to select which NIC the IP alias you specify in the haresources file will be added by comparing the IP alias to each of the destination addresses listed in your kernel's network routing table. As described in Chapter 2, you can view this routing table with the route -n command. For example:

 #route -n Kernel IP routing table Destination      Gateway        Genmask          Flags    Metric    Ref    Use   Iface 200.100.100.2    0.0.0.0        255.255.255.0    U        0         0      0     eth1 10.1.1.0         0.0.0.0        255.255.255.0    U        0         0      0     eth2 127.0.0.0        0.0.0.0        255.0.0.0        U        0         0      0     lo 0.0.0.0          200.100.100.1  0.0.0.0          UG       0         0      0     eth1

Note

The output of this command is based on entries the kernel stores in the /proc/net/route file that is created each time the system boots using the route commands in the /etc/init.d/network script on a Red Hat system. See Chapter 2 for an introduction to the kernel network routing table.

When findif is able to match the network portion of the destination address in the routing table with the network portion of the IP alias from the haresources file, it returns the interface name (eth1, for example) associated with the destination address to the IPaddr script. The IPaddr script then adds the IP alias to this local interface and adds another entry to the routing table to route packets destined for the IP alias to this local interface. So the routing table, after the 209.100.100.3 IP alias is added, would look like this:

 #route -n Kernel IP routing table Destination      Gateway        Genmask          Flags    Metric    Ref    Use   Iface 200.100.100.2    0.0.0.0        255.255.255.0    U        0         0      0     eth1 200.100.100.3    0.0.0.0        255.255.255.0    U        0         0      0     eth1 10.1.1.0         0.0.0.0        255.255.255.0    U        0         0      0     eth2 127.0.0.0        0.0.0.0        255.0.0.0        U        0         0      0     lo 0.0.0.0          200.100.100.1  0.0.0.0          UG       0         0      0     eth1

The IPaddr script has executed the command route add -host 209.100.100.3dev eth1.

Finally, to complete the process of adding the IP alias to the system, the IPaddr script sends out five gratuitous ARP broadcasts to inform locally connected computers that this IP alias is now associated with this interface.

Note

If more than one entry in the routing table matched the IP alias being added, the findif program will use the metric entry in the routing table to select the interface with the fewest hops (the lowest metric).

Finding the Right Network

To compare the network portion of the IP alias to the network portion of the "destination" address entry in the routing table, findif needs to know which portion of the address represents a network and which portion represents a node. In other words, it needs to know which network mask to apply to both the destination address in the routing table and the IP alias from the haresources file before it can determine whether the network portions of the two addresses match.

The findif program will use the network mask for each entry in the routing table and apply it to both the routing table entry and the IP alias being requested to see if they match. If the two network portions of the addresses match, they are on the same network, and findif knows to add the IP alias being requested to the interface associated with this routing table entry.

The findif program will not use the default route specified in your routing table (the entry with a destination address of 0.0.0.0) unless you specify a network mask in your haresources file. For example, the entry:

 primary.mydomain.com 209.100.100.3/24 myresource

says that Heartbeat should use a network mask of 255.255.255.0. It also says that if no other entry in the routing table with its associated network mask applied to it matches this address, the default route in the routing table should be used.

However, under normal circumstances your routing table has an entry that will match your IP alias correctly without the need to consult the default route, so you will probably never need to enter the network mask in the haresources file. In the above routing table, for example, before Heartbeat added the IP alias, the first entry looked like this:

 Destination       Gateway      Genmask         Flags     Metric    Ref    Use    Iface 200.100.100.2     0.0.0.0      255.255.255.0   U         0         0      0      eth1

This entry matches the 209.100.100.3 IP alias once the 255.255.255.0 netmask from this routing table entry is applied to both addresses for the comparison (both addresses are on the 209.100.100 network). So the correct interface (eth1 in this case) is selected even though the default route was not used in the interface selection process.

Specifying a Network Interface Card

You can avoid this auto-selection process by specifying the interface Heartbeat should use in the haresources file with an entry like this:

 primary.mydomain.com 209.100.100.3/24/eth0/209.100.100.255

This entry uses the following syntax:

 primary-server IPalias/number-of-netmask-bits/interface-name/broadcast-address

Thus, in this example, the IP alias is 209.100.100.3 with a 24-bit netmask (equivalent to a network mask of 255.255.255.0) on network interface card eth0, using a broadcast address of 209.100.100.255.

To specify this as the IP alias and interface to be used for the httpd daemon, enter the following line:

 primary.mydomain.com 209.100.100.3/24/eth0/209.100.100.255 httpd

With this entry in the haresources file, Heartbeat will always use the eth0 interface for the 209.100.100.3 IP alias when adding it to the system for the httpd daemon to use.^[6]

Customizing IP Address Takeover with the iptakeover Script

If you need to modify the routing table as part of the process of taking over an IP address, then you may want to use the iptakeover script described in Chapter 7 to perform Gratuitous ARP broadcasts yourself. To use this script, add a line like the following to the haresources file.

 primary.mydomain.com iptakeover myresource

Heartbeat will then run /etc/ha.d/resource.d/iptakeover status followed by /etc/ha.d/resource.d/myresource start. This makes it possible for you to decide exactly which interface you want your IP alias to appear on, to modify the routing table if need be, and to perform the Gratuitous ARP broadcasts from a single script. However, under most circumstances this is not required and should be avoided if possible—specify an IP address in the haresources file and let Heartbeat do all of this work for you.^[7]

The Haresources File Syntax: Resources

Each line in the haresources file can contain one or more resource script names. The resources are separated by a space. Arguments can be passed to the resource scripts using two colons between the arguments. For example, if you need Heartbeat to send a special argument (let's say FILE1) to your resource script (before the word start, status or stop), you would use the syntax:

 primary.mydomain.com myresource::FILE1

Assuming you add this line to the haresources file on both the primary and the backup server, Heartbeat will run^[8] /etc/ha.d/resource.d/myresource FILE1 start when it first starts on the primary server, and then again on the backup server when the primary server fails. When the resource needs to be "released" or stopped, Heartbeat will run the script with the command /etc/ ha.d/resource.d/myresource FILE1 stop.

If we wanted to combine our iptakeover script with the myresource script and its FILE1 argument, we would use the line:

 primary.mydomain.com iptakeover myresource::FILE1

To send your resource script several arguments, enter them all on the same line after the script name with each argument separated by a pair of colons. For example, to send the myresource script the arguments FILE1, UNAME=JOHN, and L3, your haresources entry would look like this:

 primary.mydomain.com iptakeover myresource::FILE1::UNAME=JOHN::L3

Note

The haresources syntax is also documented online at http://wiki.trick.ca/linuxha/ HeartbeatResourceAgent.

Resource Groups

Until now, we have only described resources as independent entries in the haresources file. In fact, Heartbeat considers all the resources specified on a single line as one resource group. When Heartbeat wants to know if a resource group is running, it asks only the first resource specified on the line. Multiple resource groups can be added by adding additional lines to the haresources file.

For example, if the haresources file contained an entry like this:

 primary.mydomain.com iptakeover myresource::FILE1::UNAME=JOHN::L3

only the iptakeover script would be called to ask for a status when Heartbeat was determining the status of this resource group. If the line looked like this instead:

 primary.mydomain.com myresource::FILE1::UNAME=JOHN::L3 iptakeover

Heartbeat would run the following command to determine the status of this resource group (assuming the myresource script was in the /etc/ha.d/ resource.d directory):

 /etc/ha.d/resource.d/myresource FILE1 UNAME=JOHN L3 status

Note

If you need your daemon to start before the IP alias is added to your system, enter the IP address after the resource script name with an entry like this:

     primary.mydomain.com myresource IPaddr::200.100.100.3

Resource Script Arguments and Resource Groups

To combine Heartbeat's ability to send arguments to a script with its ability to create multiple resource groups, you could, for example, write one script that started both SERVICE-A and SERVICE-B based upon the argument it was passed. For example, let's call this combined resource script resAB and assume it can handle the argument SERVICE-A or SERVICE-B followed by the word start, stop, or status to control both daemons. You could then create haresources entries like this:

 primary.mydomain.com resAB::SERVICE-A primary.mydomain.com resAB::SERVICE-B

Using this haresources file entry, when Heartbeat wanted to know if these resource groups were active, it would run the commands:

 /etc/ha.d/resource.d/resAB SERVICE-A status /etc/ha.d/resource.d/resAB SERVICE-B status

and it would start the resource group by executing:

 /etc/ha.d/resource.d/resAB SERVICE-A start /etc/ha.d/resource.d/resAB SERVICE-B start

^[1]In this book, we are only concerned with Heartbeat's ability to failover a resource from a primary server to a backup server.

^[2]Recall from Chapter 6 that this is the system init directory (/etc/init.d, /etc/rc.d/init.d/, /sbin/ init.d, /usr/local/etc/rc.d or /etc/rc.d) and the Heartbeat resource directory (/etc/ha.d/resource.d).

^[3]IP aliases were introduced in Chapter 6.

^[4]GARP broadcasts were also introduced in Chapter 6.

^[5]See "Routing Packets with the Linux Kernel" in Chapter 2.

^[6]Note that the resource daemon (httpd in this case) may also need to be configured to use this interface or this IP address as well.

^[7]Also, as we'll see in Part III, you'll want to leave IP alias assignment under Heartbeat's control so Ldirectord can failover to the backup load balancer.

^[8]This assumes the script myresource is located in the /etc/ha.d/resource.d directory and not in the /etc/init.d directory—in which case the command would be /etc/rc.d/resource.d FILE1 start.