16.5 Cluster Alias Routing

16.5 Cluster Alias Routing

So a datagram has arrived at the proxy ARP master for the cluster. The next step is to deliver that datagram to the appropriate member of the cluster for final handling. The IP software on the proxy ARP master can see that the destination IP address (the IP address associated with the cluster alias) is one of the IP addresses that it really owns. If the request is for a new TCP or UDP connection, it uses a weighted and potentially round robin list of eligible target members to decide to whom to forward the request. If it is a pre-existing TCP/IP connection, it forwards to the appropriate member.

Note that it is the IP software that makes routing decisions based on the contents of the in-memory routing table. A common misperception is that the routing daemons (routed(8) and gated(8)) make the routing decisions and implement routing. The job of the routing daemons is to populate the routing table. This means that a system can potentially be configured as a router and not have a routing daemon running. Once again, the act of routing entails forwarding packets that arrive at a system but are not ultimately destined for that system. Therefore, they must be sent back out through an interface that will move the packet closer to its destination. Within the cluster, the act of forwarding packets over the CI to a target cluster member is sometimes referred to as "tunneling".

16.5.1 Alias Routing vs. IP Routing

In a cluster, the routing decisions will primarily involve cluster alias routing, not the standard IP routing. We recommend that you use a routing device or a non-clustered system to perform standard IP routing. (This will separate the two types of routing for the sake of management simplicity. See Chapter 20 for more discussion of standard IP routing.) Cluster alias routing provides a mechanism whereby the handling of incoming network service requests (telnet, ftp, NFS, etc.) can be spread across the cluster members if desired. The forwarding of packets from the proxy ARP master to other cluster members will use the CI (Cluster Interconnect), which may be a Memory Channel or LAN-based connection.

16.5.1.1 Volunteering to Do Alias Routing

The cluster member issuing the following command has volunteered for the responsibility of routing for the cluster alias named "clua_den". It has not volunteered to actually service any requests from clients for "clua_den". Note that the string "clua_den" will have to be resolvable to an IP address through the /etc/hosts file, BIND, NIS, or another mechanism in order for this sequence to work. The /etc/hosts file is the best alternative on the cluster itself, while BIND or NIS may be preferred for ease of client use. (A less attractive alternative is to use an IP address instead of a cluster alias name.)

 # cluamgr -a alias=clua_den 

Cluster members can indicate a willingness to handle routing responsibilities, and/or join in on the service handling responsibilities (i.e., become a member of the cluster alias). Note that after volunteering for routing responsibilities (and, in this case, creating the alias), no additional information has been placed in the routing table.

 # netstat -rn | wc -l         14 

16.5.1.2 Volunteering to Service (Join) an Alias

So you figure that naturally something will be added to the routing table after the cluster member asks to "join" in the service responsibilities (see the following series of commands).

 # netstat -rn | wc -l          14 

 # cluamgr -a alias=clua_den,join 

Still no change in the routing tables. The cluster alias daemon (aliasd(8)) has not been informed of the need to start processing activities for the new alias.

 # netstat -rn | wc -l          14 

The next output shows that once the daemon has been informed (through the "cluamgr –r start" command), it adds a new route to the routing table.

 # cluamgr -r start 

 # netstat -rn | wc -l          15 

 # netstat –r | grep clua_den clua_den        localhost               UH               0        0       lo0 

16.5.1.3 Proxy ARP Master

Note also that the cluster member that created the alias has a special ARP entry in its cache indicating that the entry is permanent (won't be eliminated from the cache after 20 minutes) and published (supported by gratuitous ARP). This member will serve as the proxy ARP master for this alias.

 # arp clua_den clua_den (192.168.0.129) at 00-10-64-30-12-31 permanent published 

16.5.1.4 Alias Status and Statistics

The following output displays information and statistics about all aliases known to this cluster member. The first entry is for the default cluster alias (babylon5 in this example), and the second entry is for the newly created clua_den alias.

 [molari] # cluamgr -s all Status of Cluster Alias: babylon5.dec.com netmask: 0.0.0.0 aliasid: 1 flags: 7<ENABLED,DEFAULT,IP_V4> connections rcvd from net: 5 connections forwarded: 2 connections rcvd within cluster: 7 data packets received from network: 1373 data packets forwarded within cluster: 1289 datagrams received from network: 180 datagrams forwarded within cluster: 180 datagrams received within cluster: 6 fragments received from network: 0 fragments forwarded within cluster: 0 fragments received within cluster: 0 Member Attributes: memberid: 1, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> Status of Cluster Alias: clua_den netmask: 0.0.0.0 aliasid: 2 flags: 5<ENABLED,IP_V4> connections rcvd from net: 0 connections forwarded: 0 connections rcvd within cluster: 0 data packets received from network: 0 data packets forwarded within cluster: 0 datagrams received from network: 0 datagrams forwarded within cluster: 0 datagrams received within cluster: 0 fragments received from network: 0 fragments forwarded within cluster: 0 fragments received within cluster: 0 Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=11<JOINED,ENABLED> 

The member attributes section of this output indicates that two members (of this two member cluster) have joined in on the responsibilities of servicing client requests destined for the default cluster alias (babylon5). In addition, both members have been enabled to handle routing responsibilities (although only one member will be doing the routing for a particular alias at any one time). See the selected portion of the "cluamgr –s" command below. Note that only one member is involved with the new alias (clua_den).

 # cluamgr -s all | grep -E "^Status|^[Mm]ember|^$" | uniq Status of Cluster Alias: babylon5.dec.com Member Attributes: memberid: 1, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> Status of Cluster Alias: clua_den Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=11<JOINED,ENABLED> 

The following output is from the other cluster member. Note that the only cluster alias of which it is aware is the default cluster alias (babylon5).

 [sheridan] # cluamgr -s all Status of Cluster Alias: babylon5.dec.com netmask: 0.0.0.0 aliasid: 1 flags: 7<ENABLED,DEFAULT,IP_V4> connections rcvd from net: 6 connections forwarded: 5 connections rcvd within cluster: 3 data packets received from network: 2929 data packets forwarded within cluster: 332 datagrams received from network: 99860 datagrams forwarded within cluster: 6 datagrams received within cluster: 99558 fragments received from network: 0 fragments forwarded within cluster: 0 fragments received within cluster: 0 Member Attributes: memberid: 1, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> 

The following cluamgr command makes the issuing member a router for the clua_den cluster alias.

 [sheridan] # cluamgr -a alias=clua_den 

Nothing is evident in the routing table since aliasd has not been asked to refresh its cached information.

 [sheridan] # netstat –r | grep clua_den 

The following command adds service responsibilities to this member for the alias clua_den. Note: still no routing table changes.

 [sheridan] # cluamgr -a alias=clua_den,join 

 [sheridan] # netstat –r | grep clua_den 

The clua_den alias is now reflected as being joined and enabled on both cluster members.

 # cluamgr -s clua_den | grep -E "^Status|^[Mm]ember|^$" | uniq Status of Cluster Alias: clua_den Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=1, selp=1, rpri=1 flags=11<JOINED,ENABLED> 

After the routing table information is refreshed through the "cluamgr –r start" command, the routing tables will have been updated to reflect the handling of traffic directed to the clua_den alias. The routing table indicates that traffic for clua_den can be handled by the "localhost", which is this member. Note that the routing responsibilities will not be handled on this member (at least for now), as indicated by the absence of the "permanent and published" entry in the ARP table on this member.

 [sheridan] # cluamgr -r start 

 [sheridan] # netstat -r | grep clua_den clua_den           localhost               UH             0               0    lo0 

 [sheridan] # arp clua_den arp:  clua_den:  Unknown host 

16.5.1.5 Making Aliases

Remember the last time you watched a duck glide serenely across the surface of a pond? Beneath the surface, those little webbed feet were paddling like crazy! That's the way we view cluster aliases. You have to admit that cluster aliases seem simple on the surface, but are in fact quite complex (and we have yet to grasp the full truth about the cadence of the paddling). Before we move into deeper waters with cluster aliases, we thought it might be time to step back and examine the usefulness of this cluster feature.

16.5.1.5.1 Why Use Additional Aliases?

The cluster software creates a default cluster alias whether you want one or not. The client nodes can use the default cluster alias to access the services available within the cluster. Sounds terrific, wouldn't you say? If it's so terrific, why would we ever need to create additional cluster aliases?

If your configuration will include applications whose resources involve a subset of cluster members, it makes sense to set up a separate alias that is "joined" by selected cluster members only. Or, if your clients expect to use the cluster as an NFS server, but not all cluster members can access the NFS served storage with equal performance, it may be best to set up another alias. This alias would include selected cluster members, or include all cluster members but have different selection weight (selw) values to help load balance or otherwise determine which members are used for the bulk of the NFS serving activities. Note that it may still be best to include all cluster members in the new alias, just in case the preferred members go down. Slow access is much better than no access.

Additional aliases are created and managed by the cluamgr command. Note that the contents of the /etc/clua_alias.config file will be a series (well, at least one) of cluamgr commands. The entry to create the default cluster alias is shown in the next section. The following example creates an alias, called clua_den, and arranges for the issuing cluster member to "join" the alias, and route for the alias if necessary. This command would have to be issued on other cluster members in order for them to recognize the alias also. As shown in section 16.5.1.4, the "cluamgr –s all" command provides statistics and status reports on cluster alias use.

 # cluamgr -a alias=clua_den,join 

16.5.1.6 Making Aliases Permanent

Any changes made using the cluamgr command at the command line prompt do not survive a reboot. In order to make your alias permanent, you must add entries to the /etc/clu_alias.config file. This file is a CDSL, thus allowing each member to manage its own suite of aliases. The contents of this file will be cluamgr commands. The following example shows a /etc/clu_alias.config file with just the default cluster alias being defined.

 # cat /etc/clu_alias.config /usr/sbin/cluamgr -a selw=3,selp=1,join,alias=DEFAULTALIAS 

This file is executed as a script from within the /sbin/init.d/clu_alias script at boot time. The following is an excerpt from the /sbin/init.d/clu_alias script.

The "test –s" syntax checks to see if the file exists and has a non-zero size.

 if test -s "/etc/clu_alias.config" then      /etc/clu_alias.config fi 

For more information, see the clu_alias.config(4) reference page.

16.5.1.7 Eliminating an Alias

If a cluster member needs to change its responsibilities relative to an alias that it has previously "joined", the member may "leave" the alias. This would eliminate the member's responsibility for servicing client requests through the alias but will not eliminate alias routing responsibilities. The following example shows cluster members leaving an alias.

 [molari] # cluamgr -s clua_den | grep -E "^Status|^[Mm]ember|^$" | uniq Status of Cluster Alias: clua_den Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=11< JOINED,ENABLED> memberid: 2, selw=1, selp=1, rpri=1 flags=11< JOINED,ENABLED> 

The next command shows cluster member molari requesting to "leave" the cluster alias clua_den.

 [molari] # cluamgr -a alias=clua_den,leave 

 [molari] # cluamgr -s clua_den | grep -E "^Status|^[Mm]ember|^$" | uniq Status of Cluster Alias: clua_den Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=1, selp=1, rpri=1 flags=10<ENABLED> 

The following rsh command requests that sheridan "leave" the clua_den cluster alias as well.

 # rsh sheridan /usr/sbin/cluamgr -a alias=clua_den,leave 

Note that the alias still exists and both members are still potential routers for the alias. In order to stop routing, the cluamgr –r stop command must be issued (this stops routing for all cluster aliases), or the rpri attribute must be set to zero (this stops routing for an individual alias).

 # cluamgr -s clua_den | grep -E "^Status|^[Mm]ember|^$" | uniq Status of Cluster Alias: clua_den Member Attributes: memberid: 1, selw=1, selp=1, rpri=1 flags=10<ENABLED> memberid: 2, selw=1, selp=1, rpri=1 flags=10<ENABLED> 

Based on our testing, setting the rpri value to 0 on all members for a particular alias (and then restarting the routing activities with a "cluamgr –r start" command) will leave the cluster without a designated proxy ARP master for that alias. This is contrary to the following excerpt from the cluamgr(8) reference page.

 A value of 0 is a special case. If you set rpri=0 for an alias on a member system, that system will not route cluster alias traffic into the cluster unless two conditions are met: all active members of that alias have rpri=0, and that system is elected to provide the proxy ARP response for the alias. 

Engineering has informed us that the implementation has changed (as we pointed out above), and that the reference page is currently (V5.1A) incorrect but will be corrected in the future. The part that needs to be forced is the election of the new proxy ARP master. This will certainly happen after a reboot, but you may want to force the action by manually resetting rpri to a non- zero value on the cluster member that you would like to be the new proxy ARP master and then restarting the routing activities with the "cluamgr –r start" command.

16.5.1.8 Dynamic Routing of Cluster Aliases

Now that we have all of this terrific alias information under our belts, it's time to refine our understanding of the interactions between the network clients and the cluster itself. We have discussed the role of the ARP protocol in discerning the actual target cluster member to which the request for service is sent prior to being routed within the cluster to its ultimate destination. How, though, does a request from a client that is not on the same subnet as the cluster (assuming the cluster network interfaces are on one subnet) get resolved?

Suppose a network client resides several hops away (think of a hop as a visit to a network router) from the cluster. Our wonderful MAC magic does us no good once we are off the local area network. There has to be a mechanism whereby the cluster can advertise what it has in its routing table to other routers in the local area network. Once a router picks up the advertisement, that router, in turn, advertises (through broadcasts out each of its network interfaces) what it has in its routing table, and so forth.

16.5.1.9 Dynamic Routing Daemons

In this way, the existence of routes to cluster aliases is propagated between routers. The protocol that is used is called the Routing Information Protocol (RIP). The advertisements are sent and heard by daemon processes running the RIP protocol. In a cluster, the recommended dynamic routing daemon is gated (pronounced gate-dee), although in V5.1B (or V5.1A with appropriate patch kits installed), there is support for the use of both routed (pronounced route-dee) and static routing. Thus, a client many hops away can be directed to use the appropriate network interface to send the datagram on its way toward the cluster. Once the datagram gets to a router within the same network (or subnet) as the cluster, the MAC-level activities discussed earlier in the chapter take place.

Earlier, we saw that a new entry was placed in the routing table after a "cluamgr –r start" command was issued. The cluamgr command interacts with the aliasd daemon. The aliasd daemon maintains a special version of the traditional /etc/gated.conf file called /etc/gated.conf.membern where the "n" is the cluster member's ID. Note that this file (/etc/gated.conf.membern) is not a CDSL, but yet is member-specific as indicated by the naming convention. This allows members to offer to route for a subset of the cluster's aliases. Do not edit the /etc/gated.conf.membern file by hand. It is managed automatically by aliasd.

Normally, the /sbin/init.d/clu_alias file will be used for cluster alias subsystem initiation. This boot time script executes the /etc/clu_alias.config file which contains one or more cluamgr commands to set up the default cluster alias (and potentially others).

 # cat /etc/clu_alias.config /usr/sbin/cluamgr -a selw=3,selp=1,join,alias=DEFAULTALIAS 

Use of the cluamgr command is normally limited to troubleshooting situations but is also useful for managing temporary aliases, getting alias status and statistics, restarting routing, and causing the /etc/clua_services file to be re-read. It is used in this chapter primarily for demonstration purposes.

16.5.1.10 The gated.conf.member n File

When cluster alias routing is started on a member (or restarted using "cluamgr –r start"), the gated process is started using the special cluster version of gated.conf (/etc/gated.conf.membern ), which contains RIP information describing a host route to the cluster alias. Note the command line argument passed to gated on a cluster member.

 # ps -A | grep gated | grep -v grep 1051031 ??        S         0:00.13 /usr/sbin/gated -f /etc/gated.conf.member2 

The gated process sends broadcasts out all of the system's configured network interfaces advertising a route to any cluster aliases that it is aware of and has routing responsibilities. Remember that an alias can be "joined," in which case the cluster member is willing to provide the service represented by the alias, or the alias can be routed, or both. Use the gdc(8) command to dump gated's current state into a file if you are interested in more details on gated's function (See the gdc(8) reference page for more information.)

 # cat /etc/gated.conf.member2 # This file created automatically by "aliasd", DO NOT EDIT! # # Set up a trace file in /usr/tmp traceoptions "/usr/tmp/gated.log" replace size 1000k files 2 general; # any dead interfaces will be entered here with a high (i.e. undesireable) # preference, so that gated will be compelled to put a better route # in the kernel forwarding table # interfaces {      interface all passive; }; # broadcast routes with RIP if at least one interface is healthy # # make a default route through the cluster interconnect look expensive # # If an interface has been marked as "dead" by NIFF then we # will also exclude RIP routes via the affected interface. # This closes a hole where gated doesn't qualify the interfaces # over which it hears broadcasts, and will possibly install a route # through a dead interface. # # Lastly, we place a catch-all "all nripin noripout" here so that # if an interface is brought on-line after aliasd has started, it # won't automatically get used by gated. # rip yes {     broadcast;     interface ics0 ripin ripout metricout 10;     interface tu0 ripin ripout;     interface all noripin noripout; }; # one "host" or network number per alias # # aliasd will also reroute direct routes for interfaces # marked as "dead" by NIFF by inserting a new gateway # for that route here # static {      host 192.168.0.70 interface 127.0.0.1; }; # arrange to export alias host/network routes: # # o don't publish routes learned through RIP # o publish the "static" routes given above, except for #   orphaned networks that we're routing around # o publish direct routes for virtual subnets which have real #   interfaces attached to them # # this default may have been learned from RIP (proto rip) # or it may have been statically configured (proto kernel) # we give a slight advanatge to a default route via RIP # export proto rip interface 10.0.0.2 {     proto rip {          default;     };     proto kernel {         default metric 3;     }; }; export proto rip interface 192.168.0.69 {      proto rip restrict; }; 

The following example uses the clu_ps script available through the websites associated with this book.

 # clu_ps "=-Ao pid,command" | grep -E 'PID|gated|^[-=]|^member' | grep -v grep ================================================================================ member2  (sheridan)             ps -Ao pid,command --------------------------------------------------------------------------------     PID COMMAND 1051031 /usr/sbin/gated -f /etc/gated.conf.member2 ================================================================================ member1 (molari) ps -Ao pid,command --------------------------------------------------------------------------------     PID COMMAND 539007 /usr/sbin/gated -f /etc/gated.conf.member1 ================================================================================ 

Host routes take the highest priority in a routing table. Therefore, if a router has information about a host route (a route to a specific host), a network route (a route to the network), and a default route (a route to use when there is nothing else useful in the routing table), all in the same routing table, it will always give precedence to the host route. This information becomes significant if you decide to create a cluster alias using a virtual subnet rather than the standard common subnet (see section 16.8 for an example using a virtual subnet).