21.4 Load-balancing CFS Servers

In the next release of TruCluster Server (V5.1B), tools will be included with the clustering software that automatically manage load balancing the CFS activity, but until then, we will provide tips and examples of how to manage this load-balancing act.

The reason we even care to try and balance the CFS activity is that if the member that's performing the I/O is not the CFS server, file system traffic has to go across the cluster interconnect, whether that's MEMORY CHANNEL or Ethernet. And if you get too many unfavorable combinations of cross-member I/O, or a single member with more than its fair share of CFS serving, you can wind up with a performance problem. That doesn't mean that CFS is bad, but there is a tradeoff: a little performance for the sizable benefit of the Single System Image.

To see the current CFS layout, you can use our cfs command:

 # cfs CFS Server            Mount Point                  File System          FS Type --------------------- -----===-------------------- -------------------- ------- battra                /cluster/members/member4/    battra_usr#tmp       AdvFS                          tmp battra                /cluster/members/member4/    root4_domain#root    AdvFS                          boot_partition gamera                /archive                     archive#archive      AdvFS gamera                /usr/dec                     dec#dec              AdvFS gamera                /var/spool/ftp               ftp#ftp              AdvFS gamera                /cluster/members/member3/    gamera_usr#tmp       AdvFS                          tmp gamera                /var/spool/mail              mail#mail            AdvFS gamera                /cluster/members/member3/    root3_domain#root    AdvFS                          boot_partition gamera                /usr/staff                   staff#staff          AdvFS mothra                /                            cluster_root#root    AdvFS mothra                /usr                         cluster_usr#usr      AdvFS mothra                /var                         cluster_var#var      AdvFS mothra                /cluster/members/member2/    mothra_usr#tmp       AdvFS                         tmp mothra                /cluster/members/member2/    root2_domain#root    AdvFS                         boot_partition rodan                 /usr/ris                     ris#ris              AdvFS rodan                 /cluster/members/member1/    root1_domain#root    AdvFS                         boot_partition rodan                 /usr/src                     src#src              AdvFS rodan                 /cluster/members/member1/    usr_domain#tmp       AdvFS                         tmp

Notice that this four-member cluster is configured with local /tmp.

Your CFS management could be as simple as watching the CFS serving, as displayed by SysMan Station, and manually relocating the file systems to balance out the number of file systems per member. Or you might watch the CFS statistics with something like our custom cfs command and balance them based on the cross-member file system traffic. For example, look at this output and see if, based on the recent past, the /usr/staff file system should be relocated and if so to which member.

 # cfs -s /usr/staff /usr/staff [staff#staff] (dsk13c):                 read    write    lookup    getattr    readlnk    access    other             --------- --------- --------- --------- ----------- -------- ---------     battra:         0         0        26         0         0          1        5 *   gamera:         0         0        26        27         0          1        5     mothra:        13        40        47         0         0         20       51      rodan:       225     12880        63         3         0         21       31      total:       238     12920       162        30         0         43       92

As we can see, member rodan is the heaviest reader and writer to the /usr/staff file system and it may be beneficial to relocate it from gamera to rodan (the "*" shows that gamera is the CFS server in the above cfs output). This is a good idea if you believe that past behavior is an indicator of future behavior, but as most stockbrokers will tell you, don't bet the farm on it. Make these CFS tweaks based on your knowledge of your cluster and buttressed by statistics such as those above.

Now, let's move on to the example script. Until the HP delivered solution appears, this script should make managing the CFS a little easier. The cfsldb script, provided at the book's website (see Appendix B), reads the CFSSVR configuration variable from the member-specific rc.config file. The script will relocate file systems based on what the CFSSVR variable contains. CFSSVR is a white-space delimited list of mount points the member should serve. If the file system is already served by the member, it skips it. The script also writes a log in /cluster/admin called cfsldb_<hostname>.log which is overwritten each time the program is run.

For example, to have a member serve root, /usr, and /var:

 # rcmgr set CFSSVR "/ /usr /var"

 # ./cfsldb start cfsldb: Relocating / from sheridan to molari...done! cfsldb: Relocating /usr from sheridan to molari...done! cfsldb: Relocating /var from sheridan to molari...done!

The log will look like:

 # more /cluster/admin/cfsldb_molari.log Mon Jun   3 03:35:49 EDT 2002 cfsldb:   Attempting to relocate / from sheridan to molari cfsldb:   Relocation succeeded cfsldb:   Attempting to relocate /usr from sheridan to molari cfsldb:   Relocation succeeded cfsldb:   Attempting to relocate /var from sheridan to molari cfsldb:   Relocation succeeded   Domain or filesystem name = /var   Server Name = molari   Server Status : OK   Domain or filesystem name = /usr   Server Name = molari   Server Status : OK   Domain or filesystem name = /   Server Name = molari   Server Status : OK

If you attempt to relocate a file system that cannot be relocated, you will receive an error in the log. For example, what happens if we add a file system that is on a private bus?

 # rcmgr set CFSSVR "/ /usr /var /home"

 # ./cfsldb start cfsldb: Relocating /home from sheridan to molari...failed!!! cfsldb: One or more file systems failed to relocate             See the /cluster/admin/cfsldb_molari.log for more information

 # more /cluster/admin/cfsldb_molari.log Mon Jun 3 03:49:05 EDT 2002 cfsldb: molari is already the CFS server for / cfsldb: molari is already the CFS server for /usr cfsldb: molari is already the CFS server for /var cfsldb: Attempting to relocate /home from sheridan to molari cfsmgr: The server for /home filesystem should not be relocated to molari cfsmgr: Use -f flag for force relocation   Domain or filesystem name = /home   Server Name = sheridan   Server Status : OK

Note, the other file systems don't relocate, because they are already in the right location. This is the sort of output you'll see in /var/adm/messages:

 Jun 3 03:54:44 molari vmunix: Recovering filesystem mounted at / to this node ( member id 1) Jun 3 03:54:44 molari vmunix: Recovery to this node (member id 1) complete for filesystem mounted at / Jun 3 03:54:47 molari vmunix: Recovering filesystem mounted at /usr to this nod e (member id 1) Jun 3 03:54:48 molari vmunix: Recovery to this node (member id 1) complete for filesystem mounted at /usr Jun 3 03:54:51 molari vmunix: Recovering filesystem mounted at /var to this nod e (member id 1) Jun 3 03:54:52 molari vmunix: Recovery to this node (member id 1) complete for filesystem mounted at /var

Make sure you understand your environment before implementing a script such as this because it could lead to undesired effects such as extra cluster interconnect traffic, which would reduce file system performance if you have I/O from multiple nodes (especially if that I/O pattern changes from time to time).