|< Day Day Up >|
Chapter 5. Cluster Systems Management (CSM)
In this chapter, we explain the concepts of Cluster Systems Management (CSM) used for setting up and managing a cluster of nodes running the Linux operating system on IBM pSeries hardware. More
The following topics are discussed:
This chapter is based on the functionality of the Cluster System Manager software running with SuSE Linux Enterprise Server-8 (SLES 8) on IBM pSeries hardware. At the time of writing, SLES 8 is the only supported and recommended version of operating system for pSeries Linux.
|< Day Day Up >|
|< Day Day Up >|
5.1 CSM concepts and architecture
To understand what CSM is, it is useful to understand the details of what clustering is, what types of clusters there are, and what are the features of each cluster. We cover the fundamentals in this section, and delve into details in later sections of this chapter.
5.1.1 Clustering basics
In simple terms,
is defined as two or more computers (
5.1.2 Types of clusters
Based on functionality, clusters are grouped into four major types:
These clustering groups are discussed in the following sections.
High availability (HA) clusters
The main goal of a high availability cluster is to eliminate single points of failure (SPOFs) in the design. High availability is often
HA clusters are commonly designed based on the functionality of applications and resources that are part of the cluster. Figure 5-1 shows a simple high availability cluster.
Figure 5-1. High availability cluster
A simple HA cluster consists of two servers sharing a common disk and network. One server is the primary server running active applications, and the second server is the hot standby server, which takes over resources (shared disk, network IP and applications) in case of a problem with the primary server. This type of configuration is typically referred as an active-passive type of high availability.
There is another type of configuration in which both servers in the cluster function as primaries, running active applications at the same time. In case of a failure or problem, the surviving or active node will take over the resources of the failed server. This type of configuration is referred as an active-active type of high availability.
Depending on the failover mechanism, a high availability cluster can be designed to meet complex needs of a business scenario. Failover behavior is customized in several ways to keep the resources running on takeover node; some examples of how this can be customized include fail back immediately when the failed node becomes active, or fail back later at a scheduled time so that no
For more complex applications, such as parallel databases or applications, concurrent access configurations are designed with a distributed lock type of control and availability.
High Performance Computing (HPC) clusters
HPC clusters are used in high computing environments, with two or more nodes accessing the processors in parallel combining the power of multiple nodes at the same time.
HPC clusters are normally used in scientific computing and benchmarking centers that require a lot of processing power to analyze complex mathematical and scientific solutions. More information on HPC clusters can be found in Chapter 7, "High Performance Computing case studies" on page 303.
Figure 5-2 on page 215 shows a typical high performance cluster.
Figure 5-2. High performance cluster
Virtual load balancing clusters
These clusters are frequently used in high volume Web server and e-business environments where a front-end load balancing server routes TCP traffic to back-end Web servers on common TCP ports such as port 80 (http), port 443 (ssl), 119 (nntp) and 25 (smtp). Figure 5-3 on page 216 shows a sample virtual load balancing cluster.
Figure 5-3. Virtual load balancing cluster
The inbound traffic is received on the load balancer and is dispatched immediately to an available back-end server. The availability of back-end servers, and the server to be routed
The main advantage with these clusters is that the outside world sees only one virtual IP which in
Example 5-1. Example of virtual IP load balancing
VirtualIP = 192.168.1.10 (URL = www.testsite.com) IP of load balancer ent0 interface= 192.168.1.1 IP of load balancer ent1 interface = 10.10.10.4 IP of web server1 = 10.10.10.1 IP of web server2 = 10.10.10.2 IP of web server3 = 10.10.10.3 When a client connects to www.testsite.com, it resolves to IP 192.168.1.10 and client browser loads the page directly from one of available web servers without knowing the real IP address of that particular web server.
Depending on the requirements (such as virtual IPs, ports, and applications), load balancing can become quite complex with rules,
A management cluster consists of a
CSM uses the management server as a single point of control in the management domain managing a set of servers referred as managed nodes. Using the management server, a system administrator centralizes the cluster environment. Some of the shared
Figure 5-4 on page 218 shows a CSM cluster environment.
Figure 5-4. CSM cluster
A management server consists of a single server with management tools communicating to a group of nodes across a shared LAN. The CSM architecture includes a set of components and functions that work together to form a management cluster.
The following sections
5.1.3 RSCT, RMC, and RM
Reliable Scalable Clustering Technology (RSCT) is the backbone of the Cluster Systems Management domain. Other CSM components such as event monitoring and CSM security are based on RSCT Infrastructure. RSCT and its
Reliable Scalable Cluster Technology (RSCT)
RSCT provides a monitoring environment for CSM. RSCT was primarily developed for high availability applications such as GPFS and PSSP for AIX, and it has ported to CSM.
A detailed description of RSCT is available in RSCT for Linux: Guide and Reference , SA22-7892.
Resource Monitoring and Control (RMC)
RMC is part of RSCT. RMC is a framework for providing availability monitoring to CSM. The RMC subsystem
For further information on RMC, refer to RSCT for Linux: Guide and Reference , SA22-7892.
Resource Manager (RM)
RM is a daemon that maps resources and class attributes to commands for several resources. Some of the standard resource managers available are IBM.AuditRm for system-wide audit logging, IBM.HWCtrlRM for hardware control for managed nodes, IBM.ERRM for providing actions in response to conditions, and IBM.DMSRM for managing a set of nodes and node groups.
RSCT, RMC and RM are discussed with examples and in more detail in 5.3, "CSM administration" on page 251.
5.1.4 CSM hardware control
As the name indicates, CSM hardware control allows control of the Hardware Management Console (HMC)-connected pSeries hardware from a single point. This feature lets system administrators remotely power on/off and
For further information on hardware control, including diagnostic messages, refer to CSM for Linux: Hardware Control Guide , SA22-7856.
5.1.5 Cluster File Manager (CFM)
CSM cluster file manager allows centralized repository and file management across the cluster. This is installed along with CSM server packages. On the management server, the default directory structure under /cfmroot is created at install time with sample config files.
Using CFM, file management is simplified for all common files across a management cluster. CFM uses a push mechanism to push defined files to managed nodes thorough a cron job once every 24 hours. Customizations can be made to control which file needs to be
Configuration of CFM and details are discussed in 5.3.4, "Configuration File Manager (CFM)" on page 256.
5.1.6 CSM software maintenance
The software maintenance component provides an interface to manage remote software install and updates on to the pSeries Linux nodes. RPM packages can be queried, installed, updated and removed with this tool.
SMS uses NFS to remotely mount RPM directories, dsh to distributed commands, and Autoupdate to automatically update installed software. Auto update software is not packaged with CSM and has to be installed separately. Detailed prerequisites and configuration are discussed further in "Software maintenance" on page 259.
5.1.7 CSM diagnostic probes
CSM diagnostic probes consists of a probe manager and a set of probes. It can be used to diagnose a system problem, which is helpful to identify root cause of a problem.
Detailed descriptions of the probes, and usage examples, can be found in "Diagnostic probes" on page 265.
5.1.8 CSM security
CSM security uses underlying RMC to provide a secure environment. CSM security provides authentication, secure shell, and authorization for all managed nodes in the cluster.
By default, shell security is provided using openSSH. Secure shell is used for opening a remote shell from the management server to all managed nodes using SSH-v2 protocol. The SSHD daemon is installed and started at install time on managed nodes.
Authentication is a host-based authentication (HBA) model which uses
Public keys are exchanged only between a managed node and a management server. No keys are exchanged between the managed nodes.
A public and private key pair is generated using ssh-keygen at node install time. By default, CSM security uses "dsa"-based security for generating keys. These keys cannot be used for any other remote command applications, and they are stored at the following locations:
CSM authorization is based on an access control list (ACL) file. RMC uses this control list to verify and control any command execution by a user. The ACL file is stored at /var/ct/cfg/ctrmc.acl. This file can be modified as needed to grant access control. By default, the root account has read and write access to all resource classes, and all other uses are allowed read access only.
5.1.9 Distributed shell (dsh)
CSM packages a distributed shell in the csm.dsh package and it is installed while running installms. dsh is used for most cluster distributed management functions such as simultaneous node updates, commands, queries and probes. By default,
uses ssh for Linux nodes; it can be modified to use any other remote
Example 5-2. csmconfig output
#csmconfig AddUnrecognizedNodes = 0 (no) ClusterSNum = ClusterTM = 9078-160 ExpDate = Mon Dec 15 18:59:59 2003 HeartbeatFrequency = 12 HeartbeatSensitivity = 8 MaxNumNodesInDomain = -1 (unlimited) RegSyncDelay = 1 RemoteShell = /usr/bin/ssh SetupRemoteShell = 1 (yes)
The dsh command runs concurrently on each node specified with the -w flag, or on all nodes specified in the WCOLL environment variable. Nodegroups can also be specified with the -N flag.
A sample dsh output is shown in Example 5-3 for a single node.
Example 5-3. dsh command output for a single node
#dsh -w lpar date lpar1: Wed Oct 22 17:06:51 EDT 2003
Example 5-4 shows a group of nodes part of a node group called group1.
Example 5-4. dsh command output for a node group
#dsh -N group1 date lpar3: Wed Oct 22 17:08:28 EDT 2003 lpar1: Wed Oct 22 17:08:27 EDT 2003
The dshbak command is also part of the distributed shell and is used to format dsh output. Example 5-5 shows how to use dshbak for formatting dsh output.
Example 5-5. dsh and dshbak output
#dsh -N group1 date dshbak HOST: lpar1 ----------- Wed Oct 22 17:09:03 EDT 2003 HOST: lpar3 ----------- Wed Oct 22 17:09:03 EDT 2003
|< Day Day Up >|