5.2 CSM planning, installation and configuration
| < Day Day Up > |
5.2 CSM planning, installation and configurationIn this section, we discuss planning a CSM cluster, installing and configuring CSM on management server, and configuring CSM to add and install managed nodes.
More
At a high level, planning for setting up a CSM cluster involves:
Each of these topics are discussed in detail in the
5.2.1 Planning hardwareThe following hardware requirements must be met to support CSM on the management server, nodes in the cluster and hardware control. Management server requirements
Managed nodes requirements
Hardware control requirementsAs explained in the previous section, CSM hardware control provides remote hardware functions such as powering on/off, querying power status, and remote console from the management server to managed nodes. Following are some guidelines and requirements to utilize this feature:
Important To enable hardware control, HMC open firmware/driver version must be version 3.4 or greater and HMC build level 20030410. More information about hardware control requirements can be found in CSM for Linux: Hardware Control Guide . 5.2.2 Planning softwareCSM has requirements for both IBM and non-IBM-developed software. On a pSeries Linux cluster, including the management server and all managed nodes, SuSE Linux Enterprise Server (SLES) 8 (8.1) is mandatory. The management server must have Linux installed prior to installing CSM.
Other software requirements are summarized as
IBM softwareCSM for pSeries Linux version 1.3.2 is shipped on CSM for pSeries Linux CD-ROMs. It can also be downloaded from the following site: http://techsupport.services.ibm.com/server/cluster/fixes/csmfixhome.html Following is a list of the IBM software packaged with the CSM CD-ROM:
Non-IBM softwareSuSE Linux Enterprise Server 8 (8.1). The following non-IBM software prerequisites for CSM are shipped on the base SLES CD-ROM:
Important On the management server, kernel-ppc64-2.4.19-228.ppc.rpm or greater is required. You can copy the kernel rpm from: </csminstall/Linux/SLES/8.1/ppc64/updates/kernel/kernel***.rpm> For updates, visit: http://support.suse.de/psdb Other required non-IBM softwareThe following lists other non-IBM software required to install CSM:
5.2.3 Planning networkCSM requires a management server connected to HMC, and managed nodes on network virtual local area networks (VLANs), either on the same subnet, or on a separate subnet. But to have a secure cluster, we recommend that you have a management server connected to the HMC and managed nodes on two different VLANs. That way, the management server can maintain a secure connection with HMC on a private VLAN. We also recommend that you have all managed nodes use a separate public VLAN for internode communications, so that the Management VLAN is used exclusively for management server-to-managed node traffic only, for functions such as node updates, monitoring, and so on. Figure 5-5 shows a recommended secure CSM cluster network. Figure 5-5. Secure CSM cluster network
As shown in the figure, all pSeries managed nodes are connected to HMC on an RS232 serial network. 5.2.4 Planning securityAs discussed in "CSM security" on page 220, this includes shell security, architecture, and authentication. Shell security and authentication are performed by default using ssh and host-based authentication.
The system adiministrator can switch to rsh and enable rhosts-based security, but ssh is more secure than rsh. Network security should be planned to isolate networks and protect clusters. Authorizations can be planned to determine which
5.2.5 Installing CSM on the management serverThis section discusses installing CSM on the management server. Linux must be installed on the management server prior to performing CSM install. The CSM install involves the following major steps:
Installing SuSE Linux and Linux pre-requisites
Install the designated management server with SuSE Linux 8.1. The Enterprise server install option
Installing csm.coreNow that the management server is installed and upgraded to the latest kernel and packages, CSM install can be started. csm.core is the first package to install, as this contains the executable files required for all cluster nodes, and it installs the scripts required for further installation and setting up the environment.
This can be done either by using the CSM CD-ROM directly, or by installing it from packages copied to local disk. The CSM packages can be
The CSM package can be downloaded from the following Internet site: http://techsupport.service.ibm.com/servers/cluster/fixes/clusterfixhome.html Copy and unpack the contents to a temporary directory(/tmp/csm): # cd /tmp/csm; gzip -dc csm-linux-1.3.2.0.ppc64.tar.gz tar -xvf - The ls command on /tmp/csm is shown in Example 5-6. Example 5-6. ls -l command output in /tmp/csmtotal 49047 drwxr-xr-x 3 root root 584 Oct 23 15:39 . drwxr-xr-x 3 root root 200 Oct 23 15:39 .. -rw-r--r-- 1 root root 7360 Sep 23 10:19 README -rwxr--r-- 1 root root 34140929 Oct 23 15:32 csm-linux-1.3.2.0.ppc64.tar.gz -rw-r--r-- 1 root root 496702 Sep 23 10:19 csm.client-1.3.2.0-26.ppc.rpm -rw-r--r-- 1 root root 352444 Sep 23 10:19 csm.core-1.3.2.0-26.ppc.rpm -rw-r--r-- 1 root root 67982 Sep 23 10:19 csm.diagnostics-1.3.2.0-26.ppc.rpm -rw-r--r-- 1 root root 67605 Sep 23 10:19 csm.dsh-1.3.2.0-26.ppc.rpm -rw-r--r-- 1 root root 4261877 Sep 23 10:19 csm.server-1.3.2.0-26.ppc.rpm drwxr-xr-x 2 root root 272 Oct 23 15:39 reqs -rw-r--r-- 1 root root 3786767 Sep 23 10:19 rsct.basic-2.3.1.3-0.ppc.rpm -rw-r--r-- 1 root root 5550757 Sep 23 10:19 rsct.core-2.3.1.3-0.ppc.rpm -rw-r--r-- 1 root root 950877 Sep 23 10:19 rsct.core.utils-2.3.1.3-0.ppc.rpm -rw-r--r-- 1 root root 465261 Sep 23 10:19 src-1.2.0.3-0.ppc.rpm Copy the two following requisite packages and place them in the /tmp/csm/reqs subdirectory:
The ls command on /tmp/csm/reqs subdirectory is shown in Example 5-7. Example 5-7. ls -l command output in /tmp/csm/reqs directory-rw-r--r-- 1 root root 332141 Sep 23 10:19 tftp-hpa-0.34.tgz -rw-r--r-- 1 root root 106386 Sep 23 10:19 tftp-hpa-0.34-1.ppc64.rpm -rw-r--r-- 1 root root 168690 Sep 23 10:19 conserver-7.2.4.tar.gz -rw-r--r-- 1 root root 94586 Sep 23 10:19 conserver-7.2.4-3.ppc64.rpm -rw-r--r-- 1 root root 17750534 Sep 23 10:19 IBMJava2-JRE-1.3.1-3.0.ppc.rpm drwxr-xr-x 3 root root 536 Oct 9 14:46 .. -rwxr--r-- 1 root root 92381 Oct 13 10:30 autoupdate-5.2.6-1.noarch.rpm -rwxr--r-- 1 root root 403858 Oct 17 14:02 openCIMOM-0.7-4.noarch.rpm drwxr-xr-x 2 root root 368 Oct 17 15:29 . Install the csm.core package as follows: # rpm -i /tmp/csm/csm.core-* Or, if installing directly from CD-ROM: # mount /dev/cdrom /mnt/cdrom; rpm -i /mnt/cdrom/csm.core-* csm.core install creates and installs binaries and files in the /opt/csm directory Create /csminstall partitionWe recommend that you create a separate /csminstall partition either on LVM or on your local disk. This partition holds all necessary CSM directories and files required to manage the cluster. We created the partition on an LVM and the method is explained below.
Running installms to install the CSM server and prerequisitesNow that the csm.core package is installed, the remaining CSM packages are installed using installms script. The following steps are required before running installms:
Update PATH and MANPATH
The csm.core package installs executable files in /opt/csm/bin and csm man pages in /opt/csm/man. In order to have CSM binaries executed from
Example 5-8. PATH and MANPATH variablesexport PATH=$PATH:/opt/csm/bin export MANPATH=$MANPATH:/opt/csm/man Log out and relogin or re-execute root's .profile to force the two new variables. Verify that the variables are updated by using both of these commands: # echo $PATH # echo $MANPATH Copy the contents of the SuSE CD-ROM to local diskinstallms requires you to access SuSE base images to build /csminstall build. This is required while installing nodes using CSM and autoyast. The CD images can either be copied or specified while running installms. The images are copied to local disk first and the directory is given as THE input path while running installms. To copy THE SuSE CD image onto a local disk: # mount /dev/cdrom/mnt/cdrom;cd cdrom; tar cvf- (cd /install/sles;tar xvpf -)
Note Make sure you have sufficient disk space in the root (/) file system before copying the SuSE CD images. Running installmsThe remaining CSM installation is carried out by the installms script. The installms usage is shown in Example 5-9. Example 5-9. installms usage
Usage: installms {-p <pkg_path> -x} [-c] [-f] [-v -V] [-h]
[Attr=value [Attr=value...]]
-p <pkg_path> List of directories containing packages (p1:p2:p3)
-c Force copy from the path
-f Force install
-x Do not copy packages.
-v -V Verbose mode
-h Display usage information
Valid attributes (see the installms man page for details):
RemoteShell SetupRemoteShell
Start the installation: # /opt/csm/bin/installms -v -p /tmp/csm:/install/sles If the SuSE CD-ROM is not copied to local disk and CSM is copied to /tmp/csm, run installms as follow: # /opt/csm/bin/installms -p /tmp/csm:/media/cdrom installms prompts you to swap CD1 and 2 or the Supplementary CD1 as necessary during the install. Example 5-10 shows the output of the installms command run with both CSM and all requisites copied to local disk. Example 5-10. Example installms console output
Output log for installms is being written to /var/log/csm/installms.log.
Logging started
./installms -v -p /tmp/csm:/install/sles
Detected Linux distribution SLES 8.1.
Detected CSM distribution "CSM1.3.2".
Creating the required directories for CSM.
Searching for CSM, RSCT and Director rpms.
Searching for Open Source rpms.
Installing LINUX OS PRE REQUISITIES RPMs.
dhcp-base is already installed.
rsh-server is already installed.
perl is already installed.
dhcp-server is already installed.
pdksh is already installed.
rsync is already installed.
nfs-utils is already installed.
rdist is already installed.
tcl is already installed.
tk is already installed.
Installing expect-5.34-154.ppc.rpm RPMs.
Installing termcap-2.0.8-513.ppc.rpm RPMs.
Installing perl-XML-RegExp-0.03-295.ppc.rpm RPMs.
Installing OPEN SOURCE PRE REQUISITIES RPMs.
Installing conserver
*******************************************************************************
*
CSM has detected that the following tftp rpm is installed:
tftp-0.29-44
CSM ships with tftp-hpa-0.34-1 by default.
The above tftp rpms may conflict with
tftp-hpa-0.34-1.
You have several options at this point:
1. Remove the above tftp rpms and install tftp-hpa-0.34-1
2. Install tftp-hpa-0.34-1 without removing already installed rpms
(note: There may be conflicting files with this option, and
tftp-hpa-0.34-1 may not install correctly.)
3. Keep the above tftp rpms. Continue without installing tftp-hpa-0.34-1
4. Quit
Please choose 1, 2, 3, or 4:
1
Uninstalling tftp-0.29-44.
Installing tftp-hpa-0.34-1.ppc64.rpm RPMs.
IBMJava2-JRE is already installed.
Installing openCIMOM-0.7-4.aix5.1.noarch.rpm RPMs.
Installing RSCT RPMs.
Installing src-1.2.0.3-0.ppc.rpm RPMs.
Installing rsct.core.utils-2.3.1.3-0.ppc.rpm RPMs.
Installing rsct.core-2.3.1.3-0.ppc.rpm RPMs.
Installing CSM RPMs.
csm.core is already installed.
Installing csm.dsh-1.3.2.0-26.ppc.rpm RPMs.
Installing csm.server-1.3.2.0-26.ppc.rpm RPMs.
Installing csm.diagnostics-1.3.2.0-26.ppc.rpm RPMs.
Creating default tftpd user for the tftp server.
Creating default tftpd group for the tftp server.
About to copy CSM command binaries.
Securing file permissions in the /tftpboot directory and all subdirectories...
Installation of CSM has successfully completed!
Logging stopped
The complete installms log is written to the /var/log/csm/installms.log file. 5.2.6 Installing the CSM licenseThe CSM license is available in two options: a 60-day try and buy license available with the CSM products package, and a full production license and key shipped in a CD-ROM when purchased. Once installms installation is complete, the try and buy license is accepted by running: # csmconfig -L
The status and Expiration date can be checked with the
csmconfig
command. The Expiration date attribute (ExpDate) is
Example 5-11 on page 234 shows csmconfig command output. Example 5-11. csmconfig command output
AddUnrecognizedNodes = 0 (no)
ClusterSNum =
ClusterTM = 9078-160
ExpDate = Mon Dec 15 18:59:59 2003
HeartbeatFrequency = 12
HeartbeatSensitivity = 8
MaxNumNodesInDomain = -1 (unlimited)
RegSyncDelay = 1
RemoteShell = /usr/bin/ssh
SetupRemoteShell = 1 (yes)
If a full production license is purchased, it is accepted by running: # csmconfig -L /mnt/cdrom/csmlum.full At the prompt, follow the directions to accept the license. Again, the csmconfig command shows the status of the license and the ExpDate Attribute shows blank. 5.2.7 Install verificationEnsure that the management server is installed correctly and is ready for use before starting any configuration. Check the following to verify installation:
5.2.8 CSM management server configurationThe management server requires very minimal configuration after the installation and is basically ready for use once installms and the install verification steps outlined in 5.2.7, "Install verification" on page 234 are successfully completed. To display current configuration and modify any attributes, enter the csmconfig command; the output is shown in Example 5-15. Example 5-15. csmconfig command outputAddUnrecognizedNodes = 0 (no) ClusterSNum = ClusterTM = 9078-160 ExpDate = Mon Dec 15 18:59:59 2003 HeartbeatFrequency = 12 HeartbeatSensitivity = 8 MaxNumNodesInDomain = -1 (unlimited) RegSyncDelay = 1 RemoteShell = /usr/bin/ssh SetupRemoteShell = 1 (yes)
The attributes are
AddUnrecognizedNodes
A default value of -0- (no) indicates that the management server should not accept
ClusterSNumThe Cluster Serial number attribute is used for service. ClusterTMThe Cluster type and model is used for service and is in the form of: ####-###. ExpDateThis is the Expiration date for the try and buy license. This is blank when a Full license is accepted with the -L option. HeartbeatFrequencyThis refers to the number of seconds between heartbeat messages sent to a node from the management server. The valid value range is 1 to 900, and the default is 12. RMC is responsible for monitoring heartbeats using the node hostname. HeartbeatSensitivityThis is the number of missed heartbeat messages sent to a node to declare the node unreachable. The valid value range is 2 to 100, and the default is 8. The Node status attribute is changed from 1 to 0 when the node is unreachable. MaxNumNodesInDomainThis is the maximum number of nodes allowed in the CSM cluster. This is determined by license key. RegSyncDelayThis refers to the delay in seconds from when cluster data is updated in memory to when it is written on disk. RemoteShellThis is the path of the remote shell that CSM uses to communicate with nodes. The default is ssh. SetupRemoteShellThis indicates whether CSM can set remote shell security automatically on remote node at installtime. The default is 1 (yes). Adding Cluster SerialNumberModify the cluster serial number by entering: # csmconfig -s abc1000 5.2.9 Defining/adding nodes to the CSM clusterIn this section, we describe how to define and add cluster-managed nodes to the CSM database, and what attributes are stored for each node. Managed nodes can be either installed with CSM (which installs the operating system (Linux) and all required CSM packages), or the operating system can be pre-installed and CSM used exclusively to install CSM packages only. We will discuss both these options.
Defining and adding nodes includes these
Defining a Hardware Control PointFor pSeries standalone servers and LPARs managed by the Hardware Management Console (HMC), CSM communicates directly with the HMC by using an attribute called the Hardware Control Point.
Prior to using any hardware control, the IP address of the HMC,
Example 5-16. Example of systemid
# systemid hmc_hostname hscroot
Password: ******
Verifying, please re-enter password:******
systemid: Entry updated.
Wherein hmc_hostname is the resolvable hostname of HMC and hscroot is the userid to connect to the HMC. The userid and encrypted password are stored in the /etc/opt/system_config directory on the management server. For further information, refer to the man page of systemid. Updating /etc/hosts or name resolution
A standard hostname resolution must be implemented before attempting to define nodes. This can be either /etc/hosts for local, or /etc/resolv.conf for Domain Name Server (DNS) resolution. All the managed nodes, hardware control point and management server are resolvable for an internal cluster and managed VLANs as
If planning a DNS named implementation, refer to Chapter 3, "System administration" on page 95, for detailed information. Generating node information - creating a node definitions fileNode information can be generated for HMC-attached nodes automatically. This data can be used to create a node definition file for multiple nodes very easily. Note : For non-HMC-attached nodes, the file has to be created manually. Here, we discuss HMC-attached nodes. Node information is gathered using the command lshwinfo . Sample command and output are shown in Example 5-17. Example 5-17. lshwinfo output# lshwinfo -p hmc -c hmc_hostname -o /tmp/nodes.dat #Hostname::PowerMethod::HWControlPoint::NodeId::LParId::HWType::HWModel::HWSeri alNum no_hostname::hmc::hmc_hostname::lpar1::002::7038::6M2::10197CA no_hostname::hmc::hmc_hostname::lpar2::002::7038::6M2::10197BA The first column indicates no_hostname, as these nodes are not being defined in the CSM database yet. Edit and replace no_hostname with the proper hostname of each LPAR. The updated /tmp/nodes.dat file resembles Example 5-18. Example 5-18. Updated nodes.dat# Hostname::PowerMethod::HWControlPoint::NodeId::LParId::HWType::HWModel::HWSeria lNum node1::hmc::hmc_hostname::lpar1::002::7038::6M2::10197CA node2::hmc::hmc_hostname::lpar2::002::7038::6M2::10197BA All the node hostnames in the file, such as node1 and node2, must be resolvable either in local hosts file or on DNS. For more information about the lshwinfo command, refer to its man page.
Important Make sure the HMC is properly connected to the LPAR frame using an RS232 serial adapter, and ensure that the management server communicates with the HMC before running lshwinfo . This command fails if proper network connectivity does not exist. Once the datafile is generated, it is used to create a node definitions file by using the definenode command. Important node attributes are added using definenode and a definition file is created in the right format to populate the CSM database. Example 5-19 shows the creation of a node definition file. Example 5-19. Example definenode
# definenode -s -M /tmp/nodes.dat >/tmp/nodes.def
# cat /tmp/nodes.def
node1 :
ConsoleMethod=hmc
ConsoleSerialDevice=ttyS0
ConsoleServerName=hmc_hostname
HWControlNodeId=lpar1
HWControlPoint=ms_hostname
HWModel=6M2
HWSerialNum=10197AA
HWType=7038
InstallCSMVersion=1.3.2
InstallDistributionName=SLES
InstallDistributionVersion=8.1
InstallOSName=Linux
InstallPkgArchitecture=ppc64
LParID=001
ManagementServer=p630sles
PowerMethod=hmc
node2 :
ConsoleMethod=hmc
ConsoleSerialDevice=ttyS0
ConsoleServerName=hmc_hostname
HWControlNodeId=lpar2
HWControlPoint=ms_hostname
HWModel=6M2
HWSerialNum=10197AA
HWType=7038
InstallCSMVersion=1.3.2
InstallDistributionName=SLES
InstallDistributionVersion=8.1
InstallOSName=Linux
InstallPkgArchitecture=ppc64
LParID=002
ManagementServer=p630sles
PowerMethod=hmc
The
-s
option in definenode redirects the output to standard out, and the
-M
option reads from the mapping data file. For non-HMC-managed nodes, the same definition file can be manually created with similar attribute values. Refer to
IBM Cluster System Management Planning and Installation Guide
, SA22-7853 for detailed
Once the definition file is created, the CSM database is populated with the node definitions. This is done using the following command: # definenode -f /tmp/nodes.def Verify the CSM database using lsnode and lsnode -l commands.
Note definenode simply appends the definitions to the CSM database. If you have a typo and/or if you ran definenode -f multiple times, the same node definitions are added multiple times and this can fail the node install. To avoid this, verify by using the lsnode command that node definitions are accurate before progressing to the next step. If duplicate entries are found, clean up the CSM database by using the rmnode command to delete duplicate node entries. Modifying node attributesNode attributes added using definenode can be modified by using the chnode command. This command changes node attributes defined in the CSM database. 5.2.10 Preparing for node installationThe target install nodes defined and added in the CSM database are prepared using the csmsetupyast command. This command collects all configuration information from sources such as the SuSE CD-ROM and the default yast config file, and sets up the AutoYast configuration file to install the Linux nodes. However, you should first update the kernel for node installation before you run the csmsetupyast command (because some models will experience problems with the original kernel in SLES). Then you need to copy the new kernel rpm to /csminstall/Linux/SLES/8.1/ppc64/updates/kernel/. csmsetupyast will get the kernel image from the latest RPM by the number of the RPM name. The csmsetupyast command does the following:
The usage of the
csmsetupyast
command is
Example 5-20. csmsetupyast usage
Usage: csmsetupyast [-h]
csmsetupyast [-v -V] [-p pkg_path -x] [-k yastcfg_file]
[-P -a [-N node_groups] [-n node_list]]
[Attr=value [Attr=value...]]
-a Setup AutoYaST for all nodes. This cannot be used with
the -n, -N or -P flags.
-h Writes usage information to standard output
-n node_list
Provide a comma-separated list of nodes and node ranges to set
up AutoYaST for. (See the noderange man page for information
on node ranges.) This option cannot be used with the -a or -P
options.
-N node_groups
Provide a comma-separated list of node groups to set up
AutoYaST for. This cannot be used with the -a or -P flags.
-p pkg_path
Specifies one or more directories, separated by colons, that
contain copies of the SuSE/SLES CD-ROMs. The default is
/media/cdrom. This cannot be used with the -x flag.
-P Setup AutoYaST for all nodes whose Mode attribute is
"PreManaged". This cannot be used with the -a, -n or -N
flags.
-v -V Writes the verbose messages of the command to standard output.
-x Specifies to not copy SuSE/SLES disks. This cannot be used
with the -p flag.
-k yastcfg_file
Specifies a AutoYaST control file. The
default is /opt/csm/install/yastcfg.SLES<version>.xml.
Valid attributes (see the csmsetupyast man page for details):
Netmask
Gateway
Nameserver
Nameservers
Detailed information on each of the options can be found in the csmsetupyast man page Populating /csminstall
The /csminstall directory contains all the necessary information to install nodes such as configuration information, updated RPM packages, and the prerequisites required. Figure 5-6 shows the contents of /csminstall
Figure 5-6. /csminstall directory tree
Host
|
| < Day Day Up > |