19.2 Commands in a Cluster

Now that you know there are some menu-oriented tools available for system administration, let's dig a little deeper and see how Tru64 UNIX commands have been tweaked to handle cluster-wide activities. You will see cluster only commands in Chapter 21. Keep in mind that we are working in a SSI (single system image) environment, so we really shouldn't be too worried about each individual command. We'll mention a few commands while we're in the mood. OK?

19.2.1 Administration Commands

Pick your favorite administration command or system database file (ps (1), disklabel (8), df (1), du (1), vmstat (1), sysman, passwd (4), adduser (8), fuser (8), hwmgr (8), mount (8), etc.). The list is nearly endless, isn't it? When you run a command on a cluster member, it can either produce the same output no matter which cluster member you are using to run it, or it can produce member-specific results. Likewise, the files that you access (vmunix, printcap (4), inittab (4), rc.config (8), etc.) can be member-specific or cluster-wide. Note that rc.config and init (8) are discussed in Chapter 6.

Tru64 UNIX provides both mechanisms in order to present a single system interface to the application. Note that some commands have been modified to support cluster-wide requests. The following display shows output from a command (wall(1)) that has a cluster-specific option added to it (-c).

 [molari] # wall -c test ^D

 [molari] # Broadcast Message from obrien@molari.dec.com (pts/3) at 16:28 ... test

Out of curiosity, we checked to see if we could figure out what the wall command uses to get messages to other cluster members. The following output shows that the traditional wall command has been altered to use a non-traditional program called /usr/sbin/clu_wall. We can assume that most other commands that have cluster-specific options have been altered in some way to handle their responsibilities cluster wide.

 # strings $(which wall) | grep sbin/clu | sort | uniq /usr/sbin/clu_wall

19.2.2 Shutdown Command

The shutdown command is another example of a command that has been altered to handle cluster-wide activities. Note the "-c" option causes special actions to take place to handle user login attempts (/etc/nologin and /etc/nologin_hostname) while a shutdown is in progress. To prevent new members from joining a cluster that is being shut down, the /cluster/admin/.clu_shutdown_file is created and an entry is made in the file describing the member requesting the shutdown and the time (see example below). This file is also checked before starting a cluster-wide shutdown, thus preventing multiple simultaneous shutdown requests from occurring. The /cluster/admin/.clu_shutdown_log file does what the name implies. It logs information about cluster shutdowns (see the following example). Note also that init, halt (8), and reboot (8) have been modified to no longer unmount file systems.

 # cd /cluster/admin && ls -l .clu_shut* -rw-------   1 root      system       149 Jul     4      01:54      .clu_shutdown_file -rw-r--r--   1 root      system      3572 Jul     4      01:54      .clu_shutdown_log

 #cat /cluster/admin/.clu_shutdown_file BEGIN_ENTRY HOST=molari TIME=Sat Aug 17 14:25:50 2001 CMND=shutdown now MEMBER_SHUTDOWN_SEQUENCE=molari-ics0 END_ENTRY

 # date Sat Aug 17 15:22:13 EDT 2001

 # shutdown -c now

After the cluster is booted, let's login and check the shutdown file once again. Note that the log file tracks the history of cluster shutdowns, while the shutdown file tracks the last shutdown.

 # cat /cluster/admin/.clu_shutdown_file BEGIN_ENTRY HOST=molari TIME=Sat Aug 17 15:22:47 2001 CMND=shutdown now MEMBER_SHUTDOWN_SEQUENCE=molari-ics0 END_ENTRY

 # cat .clu_shutdown_log BEGIN_ENTRY HOST=molari TIME=Sat Aug 17 12:38:54 2001 CMND=shutdown now MEMBER_SHUTDOWN_SEQUENCE=molari-ics0 END_ENTRY

 BEGIN_ENTRY HOST=molari TIME=Sat Aug 17 14:25:50 2001 CMND=shutdown now MEMBER_SHUTDOWN_SEQUENCE=molari-ics0 END_ENTRY BEGIN_ENTRY HOST=molari TIME=Sat Aug 17 15:22:47 2001 CMND=shutdown now MEMBER_SHUTDOWN_SEQUENCE=molari-ics0 END_ENTRY

 # date Sat Aug 17 15:56:39 EDT 2001

19.2.3 File System Manipulation Commands

Other commands seem to work properly in a cluster without any special options. Consider the purpose of file manipulation commands such as showfdmn (8), mkfdmn (8), mount (8), etc. The CFS makes their actions inherently cluster-wide. Therefore you will see no special cluster options in file system oriented commands. Some commands (such as the passwd (1) command) use cluster-wide database files and thus become cluster-oriented indirectly.

Back up and restore activities in a cluster are similar to a standalone system's backup and restore, but since members share the file systems (for the most part) you only have to backup the cluster once and not once per member. CDSLs are treated as any other symbolic link would be treated.

19.2.4 Kill Command

The kill (1) command will work properly and signal the target process even if that process is running on another cluster member. You are probably a step ahead of me in figuring out how this could possibly work given that Process Identifiers (PIDs) are specific to a system. There has been a change in the meaning of several bits in the PID. If you ever find yourself yearning for the good old days when you could conveniently understand PIDs, the following script may help.

 # ps -Ao pid,comm | grep init 524289 init 525353 rdginit

 # ./cvtpid 524289 1 cluster pid = 524289; standalone pid = 1

 # ./cvtpid 525353 1065 cluster pid = 525353; standalone pid = 1065

 # cat cvtpid #! /usr/bin/ksh if [[ $# != 1 ]] then    print "USAGE: $0 <pid_to_convert>"    exit 1 fi sapid=$(print "$1 % (2 ^ 19 )" | bc ) print cluster pid = $1\; standalone pid = $sapid

Let's start a process on cluster member sheridan. Note the PID of 1067727.

 [sheridan] # cat & [1]      1067727

On molari, let's see if we can see the process. No. Nothing appears except my grep command itself.

 [molari] # ps -Ao pid,comm | grep 1067727 532818 grep 1067727

Or use our home cooked cluster-wide ps command called "clu_ps" (discussed later in this chapter).

 # clu_ps "=-efp 1067727" ================================================================================ member2    (sheridan)           ps    -efp 1067727 -------------------------------------------------------------------------------- UID          PID    PPID      C STIME       TTY        TIME CMD root      1067727  1067722    0.0 03:31:28 pts/1      0:00.01 cat ================================================================================

The process is not on molari, but then again we already knew that. However, this will not prevent us from killing the process running on sheridan from molari.

 [molari] # kill 1067727

It is gone!

 [sheridan] # ps -Ao pid,comm | grep 1067727 532818 grep 1067727

 [sheridan] # jobs -l [1] + 1067727 Terminated               cat &

Tru64 UNIX has altered the format of the PIDs beginning in V5.0 such that they include a cluster member identifier in bits 19:26 (See Chapter 6, section 6.5). This allows commands (such as the kill command) that reference a PID to indirectly use ICS to figure out on which member the target process lives.

Note that using the kill command on PID 1 sends a signal to the init process on the issuing member only.

Also note that a standalone system (not in a cluster) will be assigned member identifier zero. Recall that you can discern your system's cluster memberid through the following sysconfig command:

 # sysconfig -q generic memberid generic: memberid = 1

In a cluster, the lowest member identifier will be one. The ps command is an example of a command that will only display information that is specific to the member on which it is running. While we all wait expectantly for a cluster-wide version of ps, below is some output from a home cooked script that does the cluster-wide ps job. So in the meantime, this script is available from the Digital Press web site and from the BRUDEN web site (See Appendix B for the URLs). To get a "ps" from member2 using our clu_ps script:

 # ./clu_ps -m2 =========================================================================== member2    (sheridan)           ps -ef --------------------------------------------------------------------------- UID          PID       PPID     C STIME       TTY           TIME CMD root       1048576        0     0.0   Jul   04 ??       0:44.66 [kernel idle] root       1048577  1048576      0.0   Jul   04 ??      0:00.19 /sbin/init -a root       1048578  1048576      0.0   Jul   04 ??      0:00.00 [kproc_creator_da] root       1048579  1048578      0.0   Jul   04 ??      0:00.00 [icssvr_nomem_dae] ...

There are other options in this useful script. Here is the usage message to give you an idea of what it can do.

 # ./clu_ps -h Usage: ./clu_ps [-m memberid] ["=ps_command_options"]   Example:       ./clu_ps                        -- runs "ps -ef" on all members       ./clu_ps -m 1 "=-elfm"          -- runs "ps -elfm" on member1       ./clu_ps "=-eo pid,command"     -- runs "ps -eo pid,command" on all members       ./clu_ps "=aux"                 -- runs "ps aux" on all members

We wanted to show at least one run of the script where it acquired information from more than one member. The following uses the "-p" option of the ps command to retrieve some process information from both members of a two-member cluster. Enjoy!

 # ./clu_ps "=ap '524288,1048576'" ============================================================================ member1 (molari)               ps ap '524288,1048576' ----------------------------------------------------------------------------     PID TTY      S                TIME CMD 524288 ??        R <        1:04.73 [kernel idle] ============================================================================ member2 (sheridan)             ps ap '524288,1048576' ----------------------------------------------------------------------------     PID TTY      S                TIME CMD 1048576  ??      R <         0:45.09 [kernel idle] ============================================================================