5.2 CSM planning, installation and configuration

In this section, we discuss planning a CSM cluster, installing and configuring CSM on management server, and configuring CSM to add and install managed nodes.

More specifically , we discuss how and what to plan for a CSM cluster, describe supporting hardware, software, and network, give tips on installing and configuring the management server, and how to add and install managed nodes in the cluster.

At a high level, planning for setting up a CSM cluster involves:

• Planning hardware, software, network and security

• Installing and configuring the management server

• Adding/defining and installing nodes

Each of these topics are discussed in detail in the next few sections.

5.2.1 Planning hardware

The following hardware requirements must be met to support CSM on the management server, nodes in the cluster and hardware control.

Management server requirements

• Must be a pSeries machine p615, p630 or p650 running Linux. p655 (Cluster 1600) is not supported to be a management server.

• A logical partition (LPAR) can be configured as a management server, but we do not recommend using an LPAR as a management server. For further information on why an LPAR is not recommended, refer to CSM Guide for the PSSP Administrator , SG24-6953.

• At a minimum, 128 MB of memory and 120 MB of disk space is required to install CSM.

• Each copy of the operating system image (such as SuSE Linux 8) requires at least 2 GB of disk space.

• Each version of the operating system service packs or RPM updates and prerequisites requires an additional 2 GB of disk space.

Managed nodes requirements

• Any pSeries managed node, including p655, can be part of a CSM cluster. pSeries logical partitions (LPARs) are supported to be managed nodes.

  Restriction

  Nodes must be pSeries hardware only and cannot be mixed with any other xSeries or iSeries nodes in the pSeries Linux cluster at this time.

  
  

• Nodes can either be pre-installed with Linux or installed with the CSM management server. In the former case, the management server will install CSM packages and make it a managed node.

• A minimum of 128 MB of memory and 20 MB of disk space is required for CSM packages.

• A minimum of 2 GB of disk space is required for each operating system install.

  Important

  For p615, p630, p650 and p655, the required open firmware version is RG030407_GA4_ALL or greater.

  
  

Hardware control requirements

As explained in the previous section, CSM hardware control provides remote hardware functions such as powering on/off, querying power status, and remote console from the management server to managed nodes. Following are some guidelines and requirements to utilize this feature:

• The IBM Hardware Management Console (HMC) for pSeries hardware such as p650 and p630 is a must for hardware control.

• The management server must communicate with a HMC on a management LAN.

• HMC is connected to pSeries hardware using async serial adapters.

Important

To enable hardware control, HMC open firmware/driver version must be version 3.4 or greater and HMC build level 20030410.

More information about hardware control requirements can be found in CSM for Linux: Hardware Control Guide .

5.2.2 Planning software

CSM has requirements for both IBM and non-IBM-developed software. On a pSeries Linux cluster, including the management server and all managed nodes, SuSE Linux Enterprise Server (SLES) 8 (8.1) is mandatory. The management server must have Linux installed prior to installing CSM.

Other software requirements are summarized as follows for IBM and non-IBM vendors

IBM software

CSM for pSeries Linux version 1.3.2 is shipped on CSM for pSeries Linux CD-ROMs. It can also be downloaded from the following site:

http://techsupport.services.ibm.com/server/cluster/fixes/csmfixhome.html

Following is a list of the IBM software packaged with the CSM CD-ROM:

• csm. core 1.3.2

• csm.client 1.3.2

• csm.server 1.3.2

• csm.diagnostics 1.3.2

• csm.dsh 1.3.2

• csm.license

• rsct.basic 2.3.1

• rsct.core 2.3.1

• rsct.core. utils 2.3.1

• src 1.2.0.2

• IBM Java2-JRE 1.3.1

Non-IBM software

SuSE Linux Enterprise Server 8 (8.1).

The following non-IBM software prerequisites for CSM are shipped on the base SLES CD-ROM:

• dhcp-base-*

• dhcp-server-*

• expat -*

• expect-5.34-63*

• freetype2*

• gppshare-*

• nfs-utils-*

• pdksh-*

• perl-5*

• perl-XML-Parser-*

• perl-XML-RegExp*

• perl-libwww-perl*

• rdist-6.1.5-524*

• rsh-server-*

• rsync-*

• telnet-server*

• tcl-8*

• tk-8*

• xf86-*

Important

On the management server, kernel-ppc64-2.4.19-228.ppc.rpm or greater is required.

You can copy the kernel rpm from:

 </csminstall/Linux/SLES/8.1/ppc64/updates/kernel/kernel***.rpm> 

For updates, visit:

http://support.suse.de/psdb

Other required non-IBM software

The following lists other non-IBM software required to install CSM:

• conserver-7.2*, shipped with the CSM for pLinux CD-ROM

• tftp-hpa-0.34-1.ppc.rpm, shipped with the CSM for pLinux CD-ROM

• OpenCIMOM, available from the following site:

  http://www.ibm.com/servers/aix/products/aixos/linux/download.html

• Autoupdate 4.3.4 or higher, available from the following site:

  http:// freshmeat .net/projects/autoupdate

5.2.3 Planning network

CSM requires a management server connected to HMC, and managed nodes on network virtual local area networks (VLANs), either on the same subnet, or on a separate subnet. But to have a secure cluster, we recommend that you have a management server connected to the HMC and managed nodes on two different VLANs. That way, the management server can maintain a secure connection with HMC on a private VLAN.

We also recommend that you have all managed nodes use a separate public VLAN for internode communications, so that the Management VLAN is used exclusively for management server-to-managed node traffic only, for functions such as node updates, monitoring, and so on. Figure 5-5 shows a recommended secure CSM cluster network.

Figure 5-5. Secure CSM cluster network

As shown in the figure, all pSeries managed nodes are connected to HMC on an RS232 serial network.

5.2.4 Planning security

As discussed in "CSM security" on page 220, this includes shell security, architecture, and authentication. Shell security and authentication are performed by default using ssh and host-based authentication.

The system adiministrator can switch to rsh and enable rhosts-based security, but ssh is more secure than rsh. Network security should be planned to isolate networks and protect clusters. Authorizations can be planned to determine which user can have which access to resource classes.

5.2.5 Installing CSM on the management server

This section discusses installing CSM on the management server. Linux must be installed on the management server prior to performing CSM install. The CSM install involves the following major steps:

• Installing SuSE Linux and pre-requisites

• Installing csm.core fileset

• Creating /csminstall partition

• Running installms to install CSM and other prerequisites

Installing SuSE Linux and Linux pre-requisites

Install the designated management server with SuSE Linux 8.1. The Enterprise server install option installs most of the CSM requisite packages. Update the server with the latest service pack updates from SuSE. This upgrades the kernel to latest version (which was v2.4-21-83, at the time of writing).

Installing csm.core

Now that the management server is installed and upgraded to the latest kernel and packages, CSM install can be started. csm.core is the first package to install, as this contains the executable files required for all cluster nodes, and it installs the scripts required for further installation and setting up the environment.

This can be done either by using the CSM CD-ROM directly, or by installing it from packages copied to local disk. The CSM packages can be copied from CD-ROM if you ordered and received the media, or you can download the software from the Internet. The license file must be ordered from IBM separately and is shipped in a CD-ROM.

The CSM package can be downloaded from the following Internet site:

http://techsupport.service.ibm.com/servers/cluster/fixes/clusterfixhome.html

Copy and unpack the contents to a temporary directory(/tmp/csm):

 # cd /tmp/csm; gzip -dc csm-linux-1.3.2.0.ppc64.tar.gz  tar -xvf - 

The ls command on /tmp/csm is shown in Example 5-6.

Example 5-6. ls -l command output in /tmp/csm

 total 49047 drwxr-xr-x    3 root     root          584 Oct 23 15:39 . drwxr-xr-x    3 root     root          200 Oct 23 15:39 .. -rw-r--r--    1 root     root         7360 Sep 23 10:19 README -rwxr--r--    1 root     root     34140929 Oct 23 15:32 csm-linux-1.3.2.0.ppc64.tar.gz -rw-r--r--    1 root     root       496702 Sep 23 10:19 csm.client-1.3.2.0-26.ppc.rpm -rw-r--r--    1 root     root       352444 Sep 23 10:19 csm.core-1.3.2.0-26.ppc.rpm -rw-r--r--    1 root     root        67982 Sep 23 10:19 csm.diagnostics-1.3.2.0-26.ppc.rpm -rw-r--r--    1 root     root        67605 Sep 23 10:19 csm.dsh-1.3.2.0-26.ppc.rpm -rw-r--r--    1 root     root      4261877 Sep 23 10:19 csm.server-1.3.2.0-26.ppc.rpm drwxr-xr-x    2 root     root          272 Oct 23 15:39 reqs -rw-r--r--    1 root     root      3786767 Sep 23 10:19 rsct.basic-2.3.1.3-0.ppc.rpm -rw-r--r--    1 root     root      5550757 Sep 23 10:19 rsct.core-2.3.1.3-0.ppc.rpm -rw-r--r--    1 root     root       950877 Sep 23 10:19 rsct.core.utils-2.3.1.3-0.ppc.rpm -rw-r--r--    1 root     root       465261 Sep 23 10:19 src-1.2.0.3-0.ppc.rpm 

Copy the two following requisite packages and place them in the /tmp/csm/reqs subdirectory:

• OpenCIMOM (available from the following website).

  http://www-1.ibm.com/servers/aix/products/aixos/linux/download.html

• Autoupdate 4.3.4 or higher (available from the following website).

  http://freshmeat.net/projects/autoupdate

The ls command on /tmp/csm/reqs subdirectory is shown in Example 5-7.

Example 5-7. ls -l command output in /tmp/csm/reqs directory

 -rw-r--r--     1 root    root      332141 Sep 23 10:19 tftp-hpa-0.34.tgz -rw-r--r--     1 root    root      106386 Sep 23 10:19 tftp-hpa-0.34-1.ppc64.rpm -rw-r--r--     1 root    root      168690 Sep 23 10:19 conserver-7.2.4.tar.gz -rw-r--r--     1 root    root       94586 Sep 23 10:19 conserver-7.2.4-3.ppc64.rpm -rw-r--r--     1 root    root    17750534 Sep 23 10:19 IBMJava2-JRE-1.3.1-3.0.ppc.rpm drwxr-xr-x     3 root    root         536 Oct  9 14:46 .. -rwxr--r--     1 root    root       92381 Oct 13 10:30 autoupdate-5.2.6-1.noarch.rpm -rwxr--r--     1 root    root      403858 Oct 17 14:02 openCIMOM-0.7-4.noarch.rpm drwxr-xr-x     2 root    root         368 Oct 17 15:29 . 

Install the csm.core package as follows:

 # rpm -i /tmp/csm/csm.core-* 

Or, if installing directly from CD-ROM:

 # mount /dev/cdrom /mnt/cdrom; rpm -i /mnt/cdrom/csm.core-* 

csm.core install creates and installs binaries and files in the /opt/csm directory

Create /csminstall partition

We recommend that you create a separate /csminstall partition either on LVM or on your local disk. This partition holds all necessary CSM directories and files required to manage the cluster. We created the partition on an LVM and the method is explained below.

• Create and initialize a partition for LVM use on two SCSI disks:

  # pvcreate /dev/sda4/dev/sda2

• Create a volumegroup on your LVM partition:

  # vgcreate -v system /dev/sda4

• Create a local volume in the system volume group :

  # lvcreate -- size 3G -- name csmlv /dev/system

• Create a resiserfs file system on the logical volume:

  # mkresiserfs /dev/system/csmlv

• Create /csminstall and mount csmlv:

  # mkdir /csminstall; mount /dev/system/csmlv /csminstall

• The logical volume size can be increased or decreased on the fly (online) using lvextend and resize_resiserfs etc.

• The logical volumes and volume groups can be viewed / verified using lvdisplay, vgdisplay commands.

Running installms to install the CSM server and prerequisites

Now that the csm.core package is installed, the remaining CSM packages are installed using installms script. The following steps are required before running installms:

• Update PATH and MANPATH.

• Copy SuSE Linux Operating System CDs to local disk or have the CD-ROMs ready.

• Run installms.

Update PATH and MANPATH

The csm.core package installs executable files in /opt/csm/bin and csm man pages in /opt/csm/man. In order to have CSM binaries executed from anywhere , update root's .profile to include these two directories for PATH and MANPATH variables , respectively. Update the bash_profile file in root's home directory and edit or add the following two lines as shown in Example 5-8:

Example 5-8. PATH and MANPATH variables

 export PATH=$PATH:/opt/csm/bin export MANPATH=$MANPATH:/opt/csm/man 

Log out and relogin or re-execute root's .profile to force the two new variables. Verify that the variables are updated by using both of these commands:

 # echo $PATH # echo $MANPATH 

Copy the contents of the SuSE CD-ROM to local disk

installms requires you to access SuSE base images to build /csminstall build. This is required while installing nodes using CSM and autoyast. The CD images can either be copied or specified while running installms. The images are copied to local disk first and the directory is given as THE input path while running installms.

To copy THE SuSE CD image onto a local disk:

 # mount /dev/cdrom/mnt/cdrom;cd cdrom; tar cvf- (cd /install/sles;tar xvpf -) 

Note

Make sure you have sufficient disk space in the root (/) file system before copying the SuSE CD images.

Running installms

The remaining CSM installation is carried out by the installms script. The installms usage is shown in Example 5-9.

Example 5-9. installms usage

 Usage: installms {-p <pkg_path>  -x} [-c] [-f] [-v  -V] [-h]                  [Attr=value [Attr=value...]]         -p <pkg_path>      List of directories containing packages (p1:p2:p3)         -c                 Force copy from the path         -f                 Force install         -x                 Do not copy packages.         -v  -V            Verbose mode         -h                 Display usage information         Valid attributes (see the installms man page for details):                 RemoteShell SetupRemoteShell 

Start the installation:

 # /opt/csm/bin/installms -v -p /tmp/csm:/install/sles 

If the SuSE CD-ROM is not copied to local disk and CSM is copied to /tmp/csm, run installms as follow:

 # /opt/csm/bin/installms -p /tmp/csm:/media/cdrom 

installms prompts you to swap CD1 and 2 or the Supplementary CD1 as necessary during the install.

Example 5-10 shows the output of the installms command run with both CSM and all requisites copied to local disk.

Example 5-10. Example installms console output

 Output log for installms is being written to /var/log/csm/installms.log. Logging started ./installms -v -p /tmp/csm:/install/sles Detected Linux distribution SLES 8.1. Detected CSM distribution "CSM1.3.2". Creating the required directories for CSM. Searching for CSM, RSCT and  Director rpms. Searching for Open Source rpms. Installing LINUX OS PRE REQUISITIES RPMs. dhcp-base is already installed. rsh-server is already installed. perl is already installed. dhcp-server is already installed. pdksh is already installed. rsync is already installed. nfs-utils is already installed. rdist is already installed. tcl is already installed. tk is already installed. Installing expect-5.34-154.ppc.rpm RPMs. Installing termcap-2.0.8-513.ppc.rpm RPMs. Installing perl-XML-RegExp-0.03-295.ppc.rpm RPMs. Installing OPEN SOURCE PRE REQUISITIES RPMs. Installing conserver ******************************************************************************* * CSM has detected that the following tftp rpm is installed:         tftp-0.29-44 CSM ships with tftp-hpa-0.34-1 by default. The above tftp rpms may conflict with tftp-hpa-0.34-1. You have several options at this point: 1. Remove the above tftp rpms and install tftp-hpa-0.34-1 2. Install tftp-hpa-0.34-1 without removing already installed rpms         (note: There may be conflicting files with this option, and         tftp-hpa-0.34-1 may not install correctly.) 3. Keep the above tftp rpms. Continue without installing tftp-hpa-0.34-1 4. Quit Please choose 1, 2, 3, or 4: 1 Uninstalling tftp-0.29-44. Installing tftp-hpa-0.34-1.ppc64.rpm RPMs. IBMJava2-JRE is already installed. Installing openCIMOM-0.7-4.aix5.1.noarch.rpm RPMs. Installing RSCT RPMs. Installing src-1.2.0.3-0.ppc.rpm RPMs. Installing rsct.core.utils-2.3.1.3-0.ppc.rpm RPMs. Installing rsct.core-2.3.1.3-0.ppc.rpm RPMs. Installing CSM RPMs. csm.core is already installed. Installing csm.dsh-1.3.2.0-26.ppc.rpm RPMs. Installing csm.server-1.3.2.0-26.ppc.rpm RPMs. Installing csm.diagnostics-1.3.2.0-26.ppc.rpm RPMs. Creating default tftpd user for the tftp server. Creating default tftpd group for the tftp server. About to copy CSM command binaries. Securing file permissions in the /tftpboot directory and all subdirectories... Installation of CSM has successfully completed! Logging stopped 

The complete installms log is written to the /var/log/csm/installms.log file.

5.2.6 Installing the CSM license

The CSM license is available in two options: a 60-day try and buy license available with the CSM products package, and a full production license and key shipped in a CD-ROM when purchased.

Once installms installation is complete, the try and buy license is accepted by running:

 # csmconfig -L 

The status and Expiration date can be checked with the csmconfig command. The Expiration date attribute (ExpDate) is populated only if it is a try and buy license.

Example 5-11 on page 234 shows csmconfig command output.

Example 5-11. csmconfig command output

 AddUnrecognizedNodes = 0 (no)  ClusterSNum =  ClusterTM = 9078-160  ExpDate = Mon Dec 15 18:59:59 2003  HeartbeatFrequency = 12  HeartbeatSensitivity = 8  MaxNumNodesInDomain = -1 (unlimited)  RegSyncDelay = 1  RemoteShell = /usr/bin/ssh  SetupRemoteShell = 1 (yes) 

If a full production license is purchased, it is accepted by running:

 # csmconfig -L /mnt/cdrom/csmlum.full 

At the prompt, follow the directions to accept the license. Again, the csmconfig command shows the status of the license and the ExpDate Attribute shows blank.

5.2.7 Install verification

Ensure that the management server is installed correctly and is ready for use before starting any configuration.

Check the following to verify installation:

• Check the /var/log/csm/installms.log for errors, and fix any that you find.

• Check the installed software:

   #rpm -qagrep csm 

  Example 5-12 shows CSM packages output.

  Example 5-12. CSM packages

   csm.dsh-1.3.2.0-26 csm.core-1.3.2.0-26 csm.diagnostics-1.3.2.0-26 csm.server-1.3.2.0-26 

   #rpm -qa  grep rsct 

  Example 5-13 shows RSCT packages output.

  Example 5-13. RSCT packages

   rsct.core.utils-2.3.1.3-0 rsct.core-2.3.1.3-0 

• Verify that the RSCT subsystems are started and running:

   # lssrc -a  grep rsct 

  Example 5-14 shows the RSCT subsystems output.

  Example 5-14. RSCT subsystems

   ctrmc             rsct             1139     active  IBM.DMSRM         rsct_rm          1268     active  IBM.ERRM          rsct_rm          1269     active  IBM.HWCTRLRM      rsct_rm          1270     active  IBM.AuditRM       rsct_rm          1309     active  ctcas             rsct             11490    active  IBM.HostRM        rsct_rm          11497    active  IBM.SensorRM      rsct_rm          11507    active  IBM.CSMAgentRM    rsct_rm                   inoperative 

• Issue the csmconfig command to ensure that CSM is running. The output should look similar to Example 5-11 on page 234.

• If any CSM software updates are not installed during installms, those can be installed now.

• The contents of /tmp/csm can now be deleted.

5.2.8 CSM management server configuration

The management server requires very minimal configuration after the installation and is basically ready for use once installms and the install verification steps outlined in 5.2.7, "Install verification" on page 234 are successfully completed.

To display current configuration and modify any attributes, enter the csmconfig command; the output is shown in Example 5-15.

Example 5-15. csmconfig command output

 AddUnrecognizedNodes = 0 (no)  ClusterSNum =  ClusterTM = 9078-160  ExpDate =  Mon  Dec 15 18:59:59  2003  HeartbeatFrequency = 12  HeartbeatSensitivity = 8  MaxNumNodesInDomain = -1 (unlimited)  RegSyncDelay = 1  RemoteShell = /usr/bin/ssh  SetupRemoteShell = 1 (yes) 

The attributes are briefly explained in this section:

AddUnrecognizedNodes

A default value of -0- (no) indicates that the management server should not accept requests from unrecognized nodes to manage them and add them to cluster.

ClusterSNum

The Cluster Serial number attribute is used for service.

ClusterTM

The Cluster type and model is used for service and is in the form of: ####-###.

ExpDate

This is the Expiration date for the try and buy license. This is blank when a Full license is accepted with the -L option.

HeartbeatFrequency

This refers to the number of seconds between heartbeat messages sent to a node from the management server. The valid value range is 1 to 900, and the default is 12. RMC is responsible for monitoring heartbeats using the node hostname.

HeartbeatSensitivity

This is the number of missed heartbeat messages sent to a node to declare the node unreachable. The valid value range is 2 to 100, and the default is 8. The Node status attribute is changed from 1 to 0 when the node is unreachable.

MaxNumNodesInDomain

This is the maximum number of nodes allowed in the CSM cluster. This is determined by license key.

RegSyncDelay

This refers to the delay in seconds from when cluster data is updated in memory to when it is written on disk.

RemoteShell

This is the path of the remote shell that CSM uses to communicate with nodes. The default is ssh.

SetupRemoteShell

This indicates whether CSM can set remote shell security automatically on remote node at installtime. The default is 1 (yes).

Adding Cluster SerialNumber

Modify the cluster serial number by entering:

 # csmconfig -s abc1000 

5.2.9 Defining/adding nodes to the CSM cluster

In this section, we describe how to define and add cluster-managed nodes to the CSM database, and what attributes are stored for each node.

Managed nodes can be either installed with CSM (which installs the operating system (Linux) and all required CSM packages), or the operating system can be pre-installed and CSM used exclusively to install CSM packages only. We will discuss both these options.

Defining and adding nodes includes these tasks :

• Defining a Hardware Control Point for nodes connected by HMC

• Updating /etc/ hosts or Name resolution

• Generating node information and creating a node definition file

• Populating CSM database with node definition file

• Modifying node attributes

Defining a Hardware Control Point

For pSeries standalone servers and LPARs managed by the Hardware Management Console (HMC), CSM communicates directly with the HMC by using an attribute called the Hardware Control Point.

Prior to using any hardware control, the IP address of the HMC, userid and valid password are defined using command systemid . The command basic syntax and result are displayed in Example 5-16.

Example 5-16. Example of systemid

 # systemid hmc_hostname hscroot Password: ****** Verifying, please re-enter password:******        systemid: Entry updated. 

Wherein hmc_hostname is the resolvable hostname of HMC and hscroot is the userid to connect to the HMC. The userid and encrypted password are stored in the /etc/opt/system_config directory on the management server. For further information, refer to the man page of systemid.

Updating /etc/hosts or name resolution

A standard hostname resolution must be implemented before attempting to define nodes. This can be either /etc/hosts for local, or /etc/resolv.conf for Domain Name Server (DNS) resolution. All the managed nodes, hardware control point and management server are resolvable for an internal cluster and managed VLANs as applicable .

If planning a DNS named implementation, refer to Chapter 3, "System administration" on page 95, for detailed information.

Generating node information - creating a node definitions file

Node information can be generated for HMC-attached nodes automatically. This data can be used to create a node definition file for multiple nodes very easily.

Note : For non-HMC-attached nodes, the file has to be created manually. Here, we discuss HMC-attached nodes.

Node information is gathered using the command lshwinfo . Sample command and output are shown in Example 5-17.

Example 5-17. lshwinfo output

 # lshwinfo -p hmc -c hmc_hostname -o /tmp/nodes.dat #Hostname::PowerMethod::HWControlPoint::NodeId::LParId::HWType::HWModel::HWSeri alNum no_hostname::hmc::hmc_hostname::lpar1::002::7038::6M2::10197CA no_hostname::hmc::hmc_hostname::lpar2::002::7038::6M2::10197BA 

The first column indicates no_hostname, as these nodes are not being defined in the CSM database yet. Edit and replace no_hostname with the proper hostname of each LPAR. The updated /tmp/nodes.dat file resembles Example 5-18.

Example 5-18. Updated nodes.dat

 # Hostname::PowerMethod::HWControlPoint::NodeId::LParId::HWType::HWModel::HWSeria lNum node1::hmc::hmc_hostname::lpar1::002::7038::6M2::10197CA node2::hmc::hmc_hostname::lpar2::002::7038::6M2::10197BA 

All the node hostnames in the file, such as node1 and node2, must be resolvable either in local hosts file or on DNS. For more information about the lshwinfo command, refer to its man page.

Important

Make sure the HMC is properly connected to the LPAR frame using an RS232 serial adapter, and ensure that the management server communicates with the HMC before running lshwinfo . This command fails if proper network connectivity does not exist.

Once the datafile is generated, it is used to create a node definitions file by using the definenode command. Important node attributes are added using definenode and a definition file is created in the right format to populate the CSM database.

Example 5-19 shows the creation of a node definition file.

Example 5-19. Example definenode

 # definenode -s -M /tmp/nodes.dat >/tmp/nodes.def # cat /tmp/nodes.def node1 :         ConsoleMethod=hmc         ConsoleSerialDevice=ttyS0         ConsoleServerName=hmc_hostname         HWControlNodeId=lpar1         HWControlPoint=ms_hostname         HWModel=6M2         HWSerialNum=10197AA         HWType=7038         InstallCSMVersion=1.3.2         InstallDistributionName=SLES         InstallDistributionVersion=8.1         InstallOSName=Linux         InstallPkgArchitecture=ppc64         LParID=001         ManagementServer=p630sles         PowerMethod=hmc node2 :         ConsoleMethod=hmc         ConsoleSerialDevice=ttyS0         ConsoleServerName=hmc_hostname         HWControlNodeId=lpar2         HWControlPoint=ms_hostname         HWModel=6M2         HWSerialNum=10197AA         HWType=7038         InstallCSMVersion=1.3.2         InstallDistributionName=SLES         InstallDistributionVersion=8.1         InstallOSName=Linux         InstallPkgArchitecture=ppc64         LParID=002         ManagementServer=p630sles         PowerMethod=hmc 

The -s option in definenode redirects the output to standard out, and the -M option reads from the mapping data file. For non-HMC-managed nodes, the same definition file can be manually created with similar attribute values. Refer to IBM Cluster System Management Planning and Installation Guide , SA22-7853 for detailed explanations of which attributes are required for manual node definition creation.

Once the definition file is created, the CSM database is populated with the node definitions. This is done using the following command:

 # definenode -f /tmp/nodes.def 

Verify the CSM database using lsnode and lsnode -l commands.

Note

definenode simply appends the definitions to the CSM database. If you have a typo and/or if you ran definenode -f multiple times, the same node definitions are added multiple times and this can fail the node install.

To avoid this, verify by using the lsnode command that node definitions are accurate before progressing to the next step. If duplicate entries are found, clean up the CSM database by using the rmnode command to delete duplicate node entries.

Modifying node attributes

Node attributes added using definenode can be modified by using the chnode command. This command changes node attributes defined in the CSM database.

5.2.10 Preparing for node installation

The target install nodes defined and added in the CSM database are prepared using the csmsetupyast command. This command collects all configuration information from sources such as the SuSE CD-ROM and the default yast config file, and sets up the AutoYast configuration file to install the Linux nodes.

However, you should first update the kernel for node installation before you run the csmsetupyast command (because some models will experience problems with the original kernel in SLES). Then you need to copy the new kernel rpm to /csminstall/Linux/SLES/8.1/ppc64/updates/kernel/. csmsetupyast will get the kernel image from the latest RPM by the number of the RPM name.

The csmsetupyast command does the following:

• It sets up and populates the /csminstall directory path by copying Linux packages from a CD-ROM or directory.

• It resolves the node hostnames to be installed, and runs the getadapters command to retrieve the MAC or hardware addresses of the node's network adapters.

• It creates the dhcpd.conf file and starts dhcpd.

• It sets up the kernel+initrd image.

• It sets up the AutoYast xml config file.

• It modifies the InstallMethod node attributes in the CSM database.

The usage of the csmsetupyast command is illustrated in Example 5-20.

Example 5-20. csmsetupyast usage

 Usage:  csmsetupyast [-h]         csmsetupyast [-v  -V] [-p pkg_path  -x] [-k yastcfg_file]                    [-P  -a  [-N node_groups] [-n node_list]]                    [Attr=value [Attr=value...]]         -a       Setup AutoYaST for all nodes.  This cannot be used with                  the -n, -N or -P flags.         -h       Writes usage information to standard output         -n  node_list                  Provide a comma-separated list of nodes and node ranges to set                  up AutoYaST for.  (See the noderange man page for information                  on node ranges.)  This option cannot be used with the -a or -P                  options.         -N  node_groups                  Provide a comma-separated list of node groups to set up                  AutoYaST for.  This cannot be used with the -a or -P flags.         -p  pkg_path                  Specifies one or more directories, separated by colons, that                  contain copies of the SuSE/SLES CD-ROMs.  The default is                  /media/cdrom.  This cannot be used with the -x flag.         -P       Setup AutoYaST for all nodes whose Mode attribute is                  "PreManaged".  This cannot be used with the -a, -n or -N flags.         -v  -V  Writes the verbose messages of the command to standard output.         -x       Specifies to not copy SuSE/SLES disks.  This cannot be used                  with the -p flag.         -k  yastcfg_file                  Specifies a AutoYaST control file. The                  default is /opt/csm/install/yastcfg.SLES<version>.xml.         Valid attributes (see the csmsetupyast man page for details):                  Netmask                  Gateway                  Nameserver                  Nameservers 

Detailed information on each of the options can be found in the csmsetupyast man page

Populating /csminstall

The /csminstall directory contains all the necessary information to install nodes such as configuration information, updated RPM packages, and the prerequisites required. Figure 5-6 shows the contents of /csminstall mainly populated during CSM installation on the management server and while running the csmsetupyast command.

Figure 5-6. /csminstall directory tree

Host names and getadapters

Host names defined in the CSM database are mapped to IP addresses from local hosts file or DNS, and some network adapter information such as MAC addresses, adapter speed, duplex setting are obtained from target install nodes.

The CSM command getadapters collects network MAC addresses of the first adapter installed on target nodes, and it can be run separately before running csmsetupyast . The getadapters command will issue dsh to the node to get the MAC address ”if the node is on and has a running operating system. Otherwise, it will power-on the node and get the MAC address from the openFirmware by performing a passing ping test of the adapter.

Network information such as MAC address, adapter type, speed, and duplex setting can also be specified in a file and passed to getadapters instead of powering off/on the nodes. The same information is written to the CSM database by using the -w option.

Network attributes obtained in any other way are also be used to modify the CSM database by using the chnode command. csmsetupyast parses the node attributes database and if any network information is found, it skips running getadapters and proceeds to next step.

Example 5-21 shows common network attributes acquired and added to the CSM node attributes database.

Example 5-21. CSM node network attributes

 InstallAdapterDuplex = half  InstallAdapterGateway =  InstallAdapterMacaddr = 00:02:55:3A:06:8C  InstallAdapterNetmask =  InstallAdapterSpeed = 100  InstallAdapterType = ent 

DHCP configuration

The management server is configured as a DHCP server to assign IPs to target nodes during netboot. All the information required is obtained during csmsetupyast and a default dhcpd.conf file is written and the dhcpd daemon is started. Refer to Chapter 3, "System administration" on page 95 for more information about DHCPD.

Example 5-22 shows a sample dhcpd.conf file written for node lpar1.

Example 5-22. /etc/dhcpd.conf file

 # This file has been automatically generated by IBM CSM # Please do not modify or erase lines that contain "# CSM" # such as the following line: # CSM VERSION 1.3.2.0 (Please do not remove this line) ddns-update-style none; shared-network CSM { option routers 192.168.100.60; subnet 192.168.100.0 netmask 255.255.255.0 {                 # CSM RANGE 192.168.100.0 (Please do not remove this line)                 default-lease-time -1;                 filename "/pxelinux.0";                 next-server 192.168.100.110; } } group { ### CSM STATIC ENTRIES (Please do not remove this line)                 host lpar1 {                         hardware ethernet 00:02:55:3A:06:8C;                         fixed-address 192.168.100.77;                         filename "/csm/yast8.1-ppc64-zImage.initrd";                         option root-path "/csminstall/Linux/SLES/8.1/ppc64/CD1";                 } 

/tftpboot updates

CSM updates the /tftpboot directory during csmsetupyast processing with the kernel and boot loader information collected from initrd and boot images. This image is used to boot the target nodes during the netboot node condition.

AutoYast XML config file

The default Yastconfig file is located at /opt/csm/install/yastcfg.SLES<version>.xml.

It contains the following information:

• Default inetd services

• Default bootloader information

• Default user and encrypted password

• RPM packages to install

• Partition information

• LVM information

This information can be modified and passed as an argument by using the -k option to csmsetupyast . If the -k option is not selected, a default XML file is used as input. This information is used to generate an AutoYast config file and is written to the /csminstall/csm/csm_version/autoyast.version/ directory for each target node with a node IP or hostname prefix.

Tips

• The Yast config file is XML-formatted. Use an XML editor such as "konqueror" or "emacs" to ensure that the modified XML file is syntax-free before passing it to csmsetupyast .

Use the following tips if you are planning to modify default partition information:

• Do not create a PrepBoot partition that is more than 8 mb in size.

• Create a / (root) partition outside of Logical Volume Manager (LVM).

Other node attributes

Other node attributes, such as InstallMethod, are modified during the csmsetupyast run. These attributes are referred during installing target nodes.

Tip

For error-free updates, check the following before running csmsetupyast :

• Run an rpower -a query to make sure all target nodes respond. If errors or <unknown> are reported , fix open CIMOM or HMC errors before proceeding. getadapters powers on/off target nodes and exits if failed.

• Run rconsole -r -t -n lpar1 to see if a serial console can be opened. Fix errors such as refreshing conserver or corrupted conserver.cf before proceeding.

• Open the HMC webSM GUI and "close all virtual terminals open" to force closure of all console windows . Sometimes csmsetupyast may exit with the error message: virtual term already open .

Issue the following command to activate csmsetupyast on nodes lpar1 and lpar3:

 # /opt/csm/bin/csmsetupyast -v -k working.xml -p /install/sles -n lpar1,lpar3 

Verify the /var/log/csm/csmsetupyast.log and lsnode -l node_name to make sure no errors are reported and that the node information is defined accurately.

5.2.11 Installing managed nodes

Once csmsetupyast is successfully completed and all the node information is defined in the CSM database, the target nodes are ready to have the operating system and CSM installed, using the installnode command.

Installnode performs the following actions:

• It reboots the node and each node broadcasts its MAC address while rebooting.

• The management server's dhcpd accepts dhcp requests and adds an arp entry to have the nodes rebooted in firmware mode.

• Auto Yast initiates and completes the node install.

• Auto Yast runs post-installation custom scripts and modifies /etc/inittab with csmfirstboot and reboots the node from the local hard drive.

• The csm boot script runs makenode on the node to install CSM and configure the node as a managed node by exchanging ssh public keys.

• Once makenode completes, the Mode attribute is changed from Installing to Managed.

When a node is defined in the database but not yet installed, the Mode attribute is set to "PreManaged". Once a node is successfully installed using CSM, this attribute is changed to "Managed", at which time CSM considers this as a managed node. If the mode is set to "MinManaged", then it remains as MinManaged even after the installation.

If the installation fails for any reason, the attribute either remains as "PreManaged"or is changed to "Installing", depending at which stage the installation failed. The Mode attribute must be either "PreManaged" or "MinManaged" in order to have installnode started.

Tip

If installation fails in the middle for any reason and another run of installnode fails immediately, check the Mode attribute for the node and run this command if it is set to "Installing":

 #chnode Mode=PreManaged -n node_name 

The usage of installnode is shown in Example 5-23:

Example 5-23. installnode usage

 Usage: installnode [-h]         installnode [-v  -V] [ -P  -a  [-N node_groups] [[-n] node_list][--file filename] ]         -a          install all nodes whose InstallMethod attribute is                     kickstart.  This flag cannot be used with the -P or -N                     flags or the node_list.         -h          Display this usage information.         [-n] node_list                     Space or comma separated list of nodes and node ranges.                     (See the noderange man page for information on node                     ranges.)  This option cannot be used with the -a or -P                     options.  Node names may be specified as positional                     arguments or preceded by the -n flag.         -N node_groups                     Comma-separated list of node groups to install.  This                     flag cannot be used with the -a or -P flags.         --file filename                    specifies a file that contains a list of nodes names.                    If the file name "-", then the list is read from stdin.                    The file can contain multiple lines and each line can have                    one or node names, separated by spaces.         -P          install all nodes whose Mode attribute is PreManaged and                     whose InstallMethod attribute is kickstart.  This flag                     cannot be used with the -a or -N flags or the node_list.         -v  -V     Verbose mode 

Note

Before running installnode, make sure to have dhcpd, nfs and tftp daemons started. If not, they can be started as:

• service dhcp start

• service nfs start

• service xinetd start

Issue the installnode command to start the install on all PreManaged nodes:

 # installnode -v -P 

Note

The installnode command also runs smsupdatenode and cfmupdatenode at the end of the install to push software maintenance packages and distribute shared common CFM files.

If you do not want this to happen during installnode processing, do not populate the /cfmroot and /csminstall/Linux/OS/version/architecture/updates directory.

Example 5-24 shows the standard output of installnode .

Example 5-24. installnode output

 #installnode -n lpar3 Resolving names in the specified node list... Running cmd: /usr/bin/lsrsrc-api -D '::' -i -s IBM.ManagedNode::"Hostname IN ('lpar3')"::Name::Hostname::ManagementServer::Install1 Output log for installnode is being written to /var/log/csm/installnode.log. Loading /opt/csm/install/pkgdefs/Linux-SLES8.1.pm. Status of nodes before the install: Running cmd: /opt/csm/bin/monitorinstall 2>&1  Node                    Mode                  Status -------------------------------------------------------------------------- lpar1                    Managed               Installed lpar3                    PreManaged            AutoYaST Post-Install Complete Nodes to install:         lpar3 Running cmd: /usr/bin/lsrsrc-api -i -s IBM.DmsCtrl::::SetupRemoteShell 2>&1 Running cmd: /usr/bin/lsrsrc-api -i -s IBM.DmsCtrl::::RemoteShell 2>&1 Running cmd: /opt/csm/csmbin/remoteshell.expect -k 2>&1 Copying OpenSSH public keys to target nodes. Running cmd: /bin/cp /root/.ssh/identity.pub /csminstall/csm/con- fig/.ssh/authorized_keys Running cmd: /bin/cp /root/.ssh/id_rsa.pub /csminstall/csm/con- fig/.ssh/authorized_keys2 Running cmd: /bin/cat /root/.ssh/id_dsa.pub >> /csminstall/csm/con- fig/.ssh/authorized_keys2 CSM_FANOUT = 16, CSM_FANOUT_DELAY = 1200 . Rebooting Node for full install: lpar3 spawn /opt/csm/bin/rconsole -t -f -n lpar3 [Enter `^Ec?' for help] ..           1 = SMS Menu                          5 = Default Boot List           6 = Stored Boot List                  8 = Open Firmware Prompt      memory      keyboard     network      scsi    speaker  ok 0 > dev / ok 0 > ls 000000d9adf0:     /ethernet@1  ok 0 > " local-mac-address" 000000d8f788 get-package-property  ok 3 > . 0  ok 2 > dump 000000d9aac0: 00 02 55 3a 06 19 :..U:..: ok 0 > boot /pci@400000000111/pci@2,2/ether- net@1:speed=100,duplex=half,bootp,192.168.100.110,,192.168.100.79,0.0.0.0 auto- yast=nfs://19 BOOTP: chosen-network-type = ethernet,Status of nodes after the install: Running cmd: /opt/csm/bin/monitorinstall 2>&1 Node                    Mode                  Status -------------------------------------------------------------------------- lpar1                   Managed               Installed lpar3                   Installing            Rebooting and Installing Node. # 

Note

installnode runs /opt/csm/csmbin/hmc_nodecond, an expect script that passes SMS menu arguments while netbooting in firmware mode. Add the following two lines to the script to run in debug mode to capture more information.

 log_user 1 exp_interval 1 

Node installation is monitored in several ways:

• The monitorinstall command is used to monitor the status of the node install. Install Status and Mode status attributes are flagged from the CSM database and shown as standard out. Issue the following command to run monitor install continuously.

   #  watch  monitorinstall 

  Sample monitorinstall is shown in Example 5-25.

  Example 5-25. monitorinstall output

   Node                  Mode               Status -------------------------------------------------------------------------- lpar1                 Managed            Installed lpar3                 Installing         AutoYaST Post-Install Complete 

• Also monitor the installation by opening a read only console window. (Do not open a read write window at the beginning, as this can interrupt the install.) Open a read only console by typing:

   # rconsole -r -t -n lpar1 

Check the man page of rconsole for a list of detailed options. This command is used only for HMC-attached pSeries servers.

Making pre-installed nodes into managed nodes

For nodes that have Linux pre-installed, and you only want to install CSM and make them managed nodes after you complete defining/adding nodes, the CSM updatenode command is used instead of installnode.

This command installs CSM only on PreManaged nodes and exchanges ssh public keys with the node. It runs makenode to mount CSM package directories NFSmounted to the target node, and installs CSM client packages. Once complete, the public keys are exchanged and the status of the node is changed from PreManaged to Managed.

Tips

• If updatenode fails to exchange keys after installing CSM client filesets , the problem could be hostname resolution for that node. Check your /etc/hosts or DNS node definitions on the management server to resolve the node's hostname.

• Run updatenode with the -k option only to have RSCT public keys exchanged between the node and the management server. This is useful if the node's host name is changed and the management server no longer communicates with the node's old hostname.

5.2.12 Node install verification

Verify successful node installation as follows:

• Check the /var/log/csm/installnode.log.

• Run monitorinstall. As shown in Example 5-25,the Status column of monitorinstall shows the node as Installed .

• Run dsh -as date to display the date on all installed nodes.

• Run lsnode -H -l node-name to see if RMC is responding for the respective managed node. Sample output is shown in Example 5-26.

Example 5-26. lsnode to find out RMC response

 #lsnode -H -l lpar1 Name = lpar1  ActiveMgtScopes = 1  Architecture = ppc64  DistributionName = SLES  DistributionVersion = 8.1  KernelVersion = 2.4.21-83-pseries64  LoadAverage = {0,0,0}  NodeNameList = {lpar1}  NumProcessors = 2  NumUsers = 1  OSName = Linux  PctRealMemFree = 95  PctTotalPgSpFree = 100  PctTotalPgSpUsed = 0  PctTotalTimeIdle = 99.8824  PctTotalTimeKernel = 0.071471  PctTotalTimeUser = 0.0461264  PctTotalTimeWait = 0  RealMemSize = 4190330880  TotalPgSpFree = 524284  TotalPgSpSize = 524284  UpTime = 1067381449  VMPgInRate = 16  VMPgOutRate = 74  VMPgSpInRate = 0  VMPgSpOutRate = 0