11.2 Adding a New Cluster Member

To add a new system or new member to a cluster, we execute the clu_add_member(8)command on an existing cluster member. The clu_add_member command does the following:

Creates directories and files for the new member in the cluster_root, cluster_usr, and cluster_var file systems.
Labels and populates a boot disk for the new member.
Configures layered product software subsets.
Builds a new kernel and copies it into place in the new member's boot_partition.
Writes a record of the session to a log file, clu_add_member.log, which is located in the /cluster/admin directory. It also writes the new member's configuration information to the .membern .cfg file (where n is the memberid of the new member) located in the same directory.
There are no software installations on the new cluster member. As this is a cluster, we only need to install software once, and this was performed when the cluster was first created.

Execute the clu_add_member command on an existing cluster member. Notice that like the clu_create command in Chapter 10, the clu_add_member command tells us what information is required and exactly what it will do to create the new cluster member.

 # /usr/sbin/clu_add_member This is the TruCluster Add Member Program You will need the following information in order to add a member to the cluster:   - Hostname   - Member ID (1-63)   - Members Votes   - Member's boot disk (for example, dsk7)   - Member's virtual cluster interconnect IP name   - Member's virtual cluster interconnect IP address   - Member's physical cluster interconnect devices   - Member's NetRAIN device name   - Member's physical cluster interconnect IP address   - Member's cluster license The program will prompt for this information, offering a default value when one is available. To accept the default value, press Return If you need help responding to a prompt, either type the word 'help' or type a question mark (?) at the prompt. The program does not begin to add the member until you answer all the prompts, and you confirm that the answers are correct.

 Adding a member involves the following steps:    Labeling the boot disk (when required)    Creating AdvFS domains    Creating additional CDSLs    Updating configuration files You then boot genvmunix from the new member's boot disk. At the first boot the new member:    Configures layered product subsets    Builds a kernel and copies it to the member's boot disk    Boots the new kernel

The first thing we must do to add a new cluster member is to acknowledge that this is what we intend to do.

 Do you want to continue adding this member? [yes]:yes

11.2.1 Enter the Hostname

In the next dialogue that we have with the clu_add_member program, we are prompted to enter the fully qualified hostname that we reserved for the new cluster member. We refer to our planning document – the Cluster Preparation Checklist and Worksheet – for what we have decided to call this system.

 Each cluster member has a hostname, which is assigned to the HOSTNAME variable in /etc/rc.config. Enter the new member's fully qualified hostname []:sheridan.dec.com Checking member's hostname: sheridan.dec.com You entered 'sheridan.dec.com' as this member's hostname. Is this name correct? [yes]:yes

11.2.2 Enter the Member ID and the Expected Vote

Next, we are prompted to enter the memberid for this new cluster member. By default, the next available memberid in sequential order is presented. In this case, it is 2; however, we can use any number up to 63. Of course, this is only if the memberid is not currently in use by another cluster member.

 The next available member ID for a cluster member is '2'. To use this default value, press Return at the prompt. A member ID is used to identify each member in a cluster. Each member must have a unique member ID, which is an integer in the range 1-63, inclusive. Enter a cluster member ID [2]:2 Checking cluster member ID: 2 You entered '2' as the member ID. Is this correct? [yes]:yes

By default, this new cluster member is given 1 vote. In this case, it makes perfect sense to accept this default value. For more detailed information on quorum and the Connection Manager, please review Chapter 17.

 By default, when the current cluster's expected votes are greater then 1, each added member is assigned 1 vote(s). Otherwise, each added member is assigned 0 (zero) votes. To use this default value, press Return at the prompt. The number of votes for a member is an integer usually 0 or 1 Enter the number of votes for this member [1]:1 Checking number of votes for this member: 1 You entered '1' as the number votes for this member. Is this correct? [yes]:yes

11.2.3 Enter the Member Boot Disk

As each individual cluster member must have its own boot device or boot_partition, we are prompted to enter the name of the disk that will be used for this new cluster member. Given that we had planned for this, we refer to our completed Cluster Preparation Checklist and Worksheet for the information on the disk that has been reserved and already properly partitioned for the boot_partition of this new cluster member^[1].

 Each member has its own boot disk, which has an associated device name; for example, 'dsk5'. Enter the device name of the member boot disk []:dsk2 Checking the member boot disk: dsk2

 The specified disk contains the required 'a', 'b', and 'h' partitions. The current partition sizes are acceptable for a member's boot disk. You can either keep the current disk partition layout or have the installation program relabel the disk. If the program relabels the disk, the new label will contain the following partitions and sizes (in blocks):   Current          New   -------          ---   a: 524288        a: 524288   b: 1572864       b: 16777216   h: 2048          h: 2048 Do you want to use the current disk partitions? [yes]:yes You entered 'dsk2' as the device name of this member's boot disk. Is this correct? [yes]:yes

As you can see, we have successfully used the custom partitioned member boot disk. For more information on the special technique that we used to do this, please refer to section 10.2.1.

11.2.4 Enter the Cluster IP Name

As we had stated in Chapters 4 and 10, with the release of TruCluster Server version 5.1A and the addition of Ethernet LAN as an alternate cluster interconnect to Memory Channel, the cluster interconnect device name was changed from "mc0" to "ics0". This was done so that the cluster interconnect device could now be virtual. This change was accompanied by the change in some default naming conventions so by default, the new cluster member's virtual cluster interconnect IP name is the hostname appended with "-ics0". Prior to TruCluster Server version 5.1A, this default cluster interconnect IP name would be the hostname appended with "-mc0". While any IP name, which is not in your DNS configuration, should work, we recommend using the default for simplicity and consistency and because it is already highly descriptive.

 Device 'ics0' is the default virtual cluster interconnect device Checking virtual cluster interconnect device: ics0 The virtual cluster interconnect IP name 'sheridan-ics0' was formed by appending '-ics0' to the system's hostname. To use this default value, press Return at the prompt. Each virtual cluster interconnect interface has a unique IP name (a hostname) associated with it. Enter the IP name for the virtual cluster interconnect [sheridan-ics0]:sheridan-ics0 Checking virtual cluster interconnect IP name: sheridan-ics0 You entered 'sheridan-ics0' as the IP name for the virtual cluster interconnect. Is this name correct? [yes]:yes

11.2.5 Enter the Cluster IP Address

In Chapter 10, the subnet 10.0.0 is used as the private network for internode communications between the cluster members. By default, the 10.0.0 subnet plus the member ID is provided as a default cluster interconnect IP address for this new member. In the interest of consistency, and since it is a reserved address per RFC 1918 (private and non-routable), using this default as the virtual cluster interconnect IP address is a good choice.

 The virtual cluster interconnect IP address '10.0.0.2' was created by replacing the last byte of the virtual cluster interconnect network address '10.0.0.0' with the previously chosen member ID '2'. To use this default value, press Return at the prompt. The virtual cluster interconnect IP address is the IP address associated with the virtual cluster interconnect IP name. (192.168.168.1 is an example of an IP address.) Enter the IP address for the virtual cluster interconnect [10.0.0.2]:10.0.0.2 Checking virtual cluster interconnect IP address: 10.0.0.2 You entered '10.0.0.2' as the IP address for the virtual cluster interconnect. Is this address correct? [yes]:yes

11.2.6 Enter the Cluster Interconnect Type^[2]

Starting with TruCluster Server version 5.1A, the clu_add_member command, like the clu_create command, gives us the option to use either Ethernet LAN or Memory Channel as the cluster interconnect.^[3] At this point in adding a new member, we are prompted to select a cluster interconnect type.

For the purposes of illustration, we are providing two examples of what the selection of a cluster interconnect type should look like. The first example is using the Memory Channel and the second is using the Ethernet LAN.

Using Memory Channel is very straightforward. We just select "mc0" as the physical cluster interconnect device, confirm the selection, and we are done.

 Device 'mc0' is the default physical cluster interconnect interface device To use this default value, press Return at the prompt. The physical cluster interconnect interface device is the name of the physical device(s) which will be used for low level cluster node communications. Examples of the physical cluster interconnect interface device name are: tu0, ee0, and nr0. Enter the physical cluster interconnect device name(s) [mc0]:mc0 Checking physical cluster interconnect interface device name(s): mc0 You entered 'mc0' as your physical cluster interconnect interface device name(s). Is this correct? [yes]:yes

When adding a new cluster member, using Ethernet LAN as a physical cluster interconnect is a more complex than using Memory Channel. First, we are prompted for the physical cluster interconnect device. In our example, we use a gigabit Ethernet LAN card – "alt0".

 The physical cluster interconnect interface device is the name of the physical device(s) which will be used for low level cluster node communications. Examples of the physical cluster interconnect interface device name are: tu0, ee0, and nr0. Enter the physical cluster interconnect device name(s) []:alt0 Would you like to enter another Ethernet device? [yes]:no Checking physical cluster interconnect interface device name(s):alt0 You entered 'alt0' as your physical cluster interconnect interface device name(s). Is this correct? [yes]:yes

We are then prompted to specify whether the interfaces should be configured in a NetRAIN virtual interface. For more information on NetRAIN, please review Chapter 9. In our example, we choose not to select anything for a NetRAIN interface device name as we have only one Ethernet device for a cluster interconnect.

 Enter a NetRAIN interface device name []:

Given that the software now knows what the physical cluster interconnect device is, we need the physical cluster IP name and address. As discussed in Chapters 4 and 10, this name is created from the word "member" appended with the memberid and "-icstcp0". This is automatically created for us.

Appending the memberid to the default private subnet 10.1.0 creates the default physical cluster IP address. In our example, this IP address would be 10.1.0.2. Again, we accept this default.

 The physical cluster interconnect IP name 'member2-icstcp0' was formed by appending '-icstcp0' to the word 'member' and the member ID. Checking physical cluster interconnect IP name: member2-icstcp0 The physical cluster interconnect IP address '10.1.0.2' was created by replacing the last byte of the physical cluster interconnect network address '10.1.0.0' with the previously chosen member ID '2'. To use this default value, press Return at the prompt. The cluster physical interconnect IP address is the IP address associated with the physical cluster interconnect IP name. (192.168.168.1 is an example of an IP address.) Enter the IP address for the physical cluster interconnect [10.1.0.2]:10.1.0.2 Checking physical cluster interconnect IP address: 10.1.0.2 You entered '10.1.0.2' as the IP address for the physical cluster interconnect. Is this address correct? [yes]:yes

For more information on configuring and using an Ethernet LAN card as a cluster interconnect, please review section 4.4.2.

11.2.7 TruCluster Server License PAK

Finally, the clu_add_member command prompts us for the last input – the TruCluster Server License PAK. With this prompt, we can choose to either add the license PAK for the new cluster member now or use the lmfsetup command later after we boot the new cluster member. In our example, we choose to enter the TruCluster Server License PAK.

 Each cluster member must have its own registered TruCluster Server license. The data required to register a new member is typically located on the License PAK certificate or it may have been previously placed on your system as a partial or complete license data file. If you are prepared to enter this license data at this time, clu_add_member can configure the new member to use this license data. If you do not have the license data at this time you can enter this data on the new member when it is up and running. Do you wish to register the TruCluster Server license for this new member at this time? [yes]:yes

 By default the TruCluster Server license data is entered by invoking the editor that is defined by your EDITOR environment variable. If the environment variable is undefined, the vi editor will be invoked. clu_add_member will display a template that includes all the fields on a PAK and an additional field for your comment. You must transfer the values from your License Pak certificate to the template, then save and exit the editor. Do you wish to enter the data using the 'vi' editor? [yes]:yes

 Licensed Software Product Authorization Key Enter data on lines terminated with :                   Issuer :dec     Authorization Number :alq-nq-2000dec30-zzzz             Product Name :tcs-ua                 Producer :dec          Number of units :1050                  Version :     Product Release Date :     Key Termination Date :  Availability Table Code :h      Activity Table Code :              Key Options :mod_units,alpha            Product Token :              Hardware-Id :                 Checksum :2-wwww-xxxx-yyyy-zzzz Creating new license database Warning creating new history file Checking TruCluster Server license data

11.2.8 Summary of Inputs

Given that we have completed all the required inputs to add a member to a cluster, we are given a chance to review and accept all that we have entered. As we had two examples using the different types of cluster interconnect devices, we have two different summaries.

This first input summary is based on using Memory Channel as a cluster interconnect.

 You entered the following information:     Member's hostname:                                    sheridan.dec.com     Member's ID:                                          2     Number of votes assigned to this member:              1     Member's boot disk:                                   dsk2     Member's virtual cluster interconnect devices:        ics0     Member's virtual cluster interconnect IP name:        sheridan-ics0     Member's virtual cluster interconnect IP address:     10.0.0.2     Member's physical cluster interconnect devices:       mc0     Member's NetRAIN device name:                         Not-Applicable     Member's physical cluster interconnect IP address:    Not-Applicable     Member's cluster license:                             Entered If you want to change any of the above information answers 'n' to the following prompt. You will then be given an opportunity to change your selections. Do you want to continue to add this member? [yes]:yes

This next input summary is based on using Ethernet LAN as a cluster interconnect.

 You entered the following information:     Member's hostname:                                    sheridan.dec.com     Member's ID:                                          2     Number of votes assigned to this member:              1     Member's boot disk:                                   dsk2     Member's virtual cluster interconnect devices:        ics0     Member's virtual cluster interconnect IP name:        sheridan-ics0     Member's virtual cluster interconnect IP address:     10.0.0.2     Member's physical cluster interconnect devices:       alt0     Member's NetRAIN device name:                         Not-Applicable     Member's physical cluster interconnect IP address:    10.1.0.2     Member's cluster license:                             Entered If you want to change any of the above information answers 'n' to the following prompt. You will then be given an opportunity to change your selections. Do you want to continue to add this member? [yes]: yes

In either example, we confirm the summary and answer "yes" when prompted to continue adding the cluster member.

11.2.9 Configuring the Cluster

This next section does not require any input as all selections and entries have already been made. Let's examine what is being done.

First, a disk label is written to the new cluster member's boot disk. The cnx partition on this disk is then initialized.

 Creating required disk labels.   Creating disk label on member disk : dsk2   Initializing cnx partition on member disk : dsk2h

The AdvFS domain for the new cluster member's boot disk is created along with member-specific files for the new cluster member.

 Creating AdvFS domains:   Creating AdvFS domain 'root2_domain#root' on partition '/dev/disk/dsk2a'. Creating cluster member-specific files:   Creating new member's root member-specific files   Creating new member's usr member-specific files   Creating new member's var member-specific files   Creating new member's boot member-specific files

Next, as part of the addition of the new cluster member, configuration modifications and updates to system files are performed. Again, as we have two different examples of configurations using the different cluster interconnect types, we also have two different examples of modifications to system files.

If a Memory Channel is used as a cluster interconnect, the updates to system files looks like this:

 Modifying configuration files required for new member operation:   Updating /etc/hosts - associating IP address '10.0.0.2' with hostname 'sheridan-ics0'   Updating /etc/rc.config   Updating /etc/rc.config   Updating /etc/sysconfigtab   Updating member-specific /etc/inittab file with 'cms' entry.   Updating /.rhosts - adding hostname 'sheridan-ics0'   Updating /etc/hosts.equiv - adding hostname 'sheridan-ics0'   Updating /etc/cfgmgr.auth - adding hostname 'sheridan.dec.com'   Configuring cluster alias.   Configuring Network Time Protocol for new member   Adding interface 'molari-ics0' as an NTP peer to member 'sheridan.dec.com'   Adding interface 'sheridan-ics0' as an NTP peer to member 'molari.dec.com'   Registering TruCluster Server License

If an Ethernet LAN is used as a cluster interconnect, the updates to system files look like this:

 Modifying configuration files required for new member operation:   Updating /etc/hosts - associating IP address '10.0.0.2' with hostname 'sheridan-ics0'   Updating /etc/hosts - adding IP address '10.1.0.2' and hostname 'member2-icstcp0'   Updating /etc/rc.config   Updating /etc/sysconfigtab   Updating member-specific /etc/inittab file with 'cms' entry.   Updating /etc/securettys - adding ptys entry   Updating /.rhosts - adding hostname 'sheridan-ics0'   Updating /etc/hosts.equiv - adding hostname 'sheridan-ics0'   Updating /.rhosts - adding hostname 'member2-icstcp0'   Updating /etc/hosts.equiv - adding hostname 'member2-icstcp0'   Updating /etc/cfgmgr.auth - adding hostname 'sheridan.dec.com'   Configuring cluster alias.   Configuring Network Time Protocol for new member   Adding interface 'molari-ics0' as an NTP peer to member 'sheridan.dec.com'   Adding interface 'sheridan-ics0' as an NTP peer to member 'molari.dec.com'   Registering TruCluster Server License

Once the entire configuration for the addition of the new cluster member is complete, a list of instructions on what to do next on the new cluster member is displayed.

Configuring automatic subset configuration and kernel build. CNX MGR: Adjust expected votes operation completed with quorum. clu_add_member: Initial member 2 configuration completed successfully. From the newly added member's console, perform the following steps to complete the newly added member's configuration: 1. Set the console variable 'boot_osflags' to 'A'. 2. Identify the console name of the newly added member's boots device. >>>show device The newly added member's boot device has the following properties: Manufacturer: DEC Model: HSG80 Target: IDENTIFIER=2 Lun: UNKNOWN Serial Number: SCSI-WWID:01000010:6000-1fe1-0001-6950-0009-1420-0755-0038 Note: The SCSI bus number may differ when viewed from different members. 3. Boot the newly added member using genvmunix: >>>boot -file genvmunix <new-member-boot-device> During this initial boot the newly added member will: o Configure each installed subset. o Attempt to build and install a new kernel. If the system cannot build a kernel, it starts a shell where you can attempt to build a kernel manually. If the build succeeds, copy the new kernel to /vmunix. When you are finished exit the shell using ^D or 'exit'. o The newly added member will attempt to set boot related console variables and continue to boot to multi-user mode. o After the newly added member boots you should setup your system default network interface using the appropriate system management command.

^[1]Please refer to section 11.1.2.2 on The New Member Boot Disk and section 10.1.2 on Disks for the Cluster.

^[2]This section of the clu_add_member is not present in TruCluster Server version 5.0A and TruCluster Server version 5.1.

^[3]TruCluster Server version 5.0A and TruCluster Server version 5.1 only support Memory Channel as a cluster interconnect.

11.2 Adding a New Cluster Member