List of Figures

---

Chapter 1: Introduction

Figure 1-1: Main architectural elements of IBM Tivoli Workload Scheduler relevant to high availability

Figure 1-2: Typical site configuration of all Tivoli products that can be integrated with IBM Tivoli Workload Scheduler out of the box

Figure 1-3: Relationship between major components of IBM Tivoli Workload Scheduler and IBM Tivoli Management Framework

Figure 1-4: Cost and benefits of availability technologies

Figure 1-5: Example disaster recovery incident where no job recovery is required

Figure 1-6: Example disaster recovery incident where job recovery not related to IBM Tivoli Workload Scheduler is required

Figure 1-7: Example disaster recovery incident requiring multiple, different job recovery actions

Figure 1-8: Standby configuration in normal operation

Figure 1-9: Standby configuration in fallover operation

Figure 1-10: Takeover configuration in normal operation

Figure 1-11: Takeover configuration in fallover operation

Figure 1-12: Example HACMP Cluster

Figure 1-13: Introductory high availability road map

Figure 1-14: Road map for implementing highly available IBM Tivoli Workload Scheduler (no IBM Tivoli Management Framework, no Job Scheduling Console access through cluster nodes)

Figure 1-15: Road map for implementing IBM Tivoli Workload Scheduler in a highly available configuration, with IBM Tivoli Management Framework

Figure 1-16: Road map for implementing IBM Tivoli Management Framework by itself

Chapter 2: High Level Design and Architecture

Figure 2-1: A typical HA cluster configuration

Figure 2-2: Resource groups in a cluster

Figure 2-3: Fallover of a resource group

Figure 2-4: Two-node cluster

Figure 2-5: Multi-node cluster

Figure 2-6: Symbol of IBM Tivoli Workload Scheduler engine availability

Figure 2-7: z/OS access method

Figure 2-8: Three-instance configuration

Figure 2-9: IBM Tivoli Workload Scheduler job flow

Chapter 3: High Availability Cluster Implementation

Figure 3-1: Overview of our HA cluster scenario

Figure 3-2: A hot standby cluster for a TMR server

Figure 3-3: Cluster node plan

Figure 3-4: Cluster diagram with applications added

Figure 3-5: Cluster diagram with network topology added

Figure 3-6: Cluster diagram with disks added

Figure 3-7: Cluster diagram with shared LVM added

Figure 3-8: Cluster diagram with resource group added

Figure 3-9: SSA Cabling for high availability scenario

Figure 3-10: Volume Group SMIT menu

Figure 3-11: Defining a volume group

Figure 3-12: Add a Journaled File System menu

Figure 3-13: Defining a journaled file system

Figure 3-14: Changing a Logical Volume menu

Figure 3-15: Renaming a logical volume

Figure 3-16: Import a Volume Group

Figure 3-17: Changing a Volume Group screen

Figure 3-18: Changing the properties of a volume group

Figure 3-19: The TCP/IP SMIT menu

Figure 3-20: Configuring network adapters

Figure 3-21: IBM Fix Central Web page for downloading AIX 5.2 maintenance packages

Figure 3-22: Using the IBM Fix Central Web page for downloading HACMP 5.1 patches

Figure 3-23: Select fixes page of IBM Fix Central Web site

Figure 3-24: Confirmation dialog presented in IBM Fix Central Select fixes page

Figure 3-25: Select fixes page showing fixes found that match APAR IY45695

Figure 3-26: Packaging options page for packaging fixes for APAR IY45695

Figure 3-27: Download fixes page for fixes related to APAR IY45695

Figure 3-28: Screen displayed after running command smitty install

Figure 3-29: Filling out the INPUT device/directory for software field in the Install Software smit panel

Figure 3-30: Install Software SMIT panel with all installation options

Figure 3-31: Installation confirmation dialog for SMIT

Figure 3-32: COMMAND STATUS SMIT panel showing successful installation of HACMP 5.1

Figure 3-33: Update Software by Fix (APAR) SMIT panel displayed by running command smitty update

Figure 3-34: Entering directory of APAR IY45695 fixes into Update Software by Fix (APAR) SMIT panel

Figure 3-35: Preparing to select fixes for APAR IY45695 in Update Software by Fix (APAR) SMIT panel

Figure 3-36: Selecting fixes for APAR IY45695 in FIXES to install SMIT dialog

Figure 3-37: Selecting all fixes of APAR IY45695 in FIXES to install SMIT dialog

Figure 3-38: Applying all fixes of APAR IY45695 in Update Software by Fix (APAR) SMIT panel

Figure 3-39: COMMAND STATUS SMIT panel showing all fixes of APAR IY45695 successfully applied

Figure 3-40: How to specify removal of HACMP in Remove Installed Software SMIT panel

Figure 3-41: Set options for removal of HACMP in Installed Software SMIT panel

Figure 3-42: Successful removal of HACMP as shown by COMMAND STATUS SMIT panel

Figure 3-43: Microsoft Cluster environment

Figure 3-44: Windows Components Wizard

Figure 3-45: Windows Components Wizard

Figure 3-46: Welcome screen

Figure 3-47: Hardware Configuration

Figure 3-48: Create or Join a Cluster

Figure 3-49: Cluster Name

Figure 3-50: Select an Account

Figure 3-51: Add or Remove Managed Disks

Figure 3-52: Cluster File Storage

Figure 3-53: Warning window

Figure 3-54: Network Connections - All communications

Figure 3-55: Network Connections - Internal cluster communications only (private network)

Figure 3-56: Network priority setup

Figure 3-57: Cluster IP Address

Figure 3-58: Cluster Service Configuration Wizard

Figure 3-59: Cluster Service Configuration Wizard

Figure 3-60: Windows Components Wizard

Figure 3-61: Windows Components Wizard

Figure 3-62: Windows Components Wizard

Figure 3-63: Windows Components Wizard

Figure 3-64: Hardware Configuration

Figure 3-65: Create or Join a Cluster

Figure 3-66: Cluster Name

Figure 3-67: Select an Account

Figure 3-68: Finish the installation

Figure 3-69: Verifying that the cluster works

Figure 3-70: Cluster resource diagram

Figure 3-71: Rename the cluster resource groups

Figure 3-72: Changing resource groups

Figure 3-73: Resource move confirmation

Figure 3-74: Rename resources

Figure 3-75: Add a new resource

Figure 3-76: Name resource and select resource type

Figure 3-77: Select resource owners

Figure 3-78: Dependency configuration

Figure 3-79: Configure IP address

Figure 3-80: Completion dialog

Figure 3-81: Adding a new resource

Figure 3-82: Specify resource name and type

Figure 3-83: Select resource owners

Figure 3-84: Resource dependency configuration

Figure 3-85: Cluster name

Figure 3-86: Completion dialog

Figure 3-87: Bring resource group online

Chapter 4: IBM Tivoli Workload Scheduler Implementation in a Cluster

Figure 4-1: IBM Tivoli Workload Scheduler implementation overview

Figure 4-2: Adding a group

Figure 4-3: Defining a user

Figure 4-4: Logging into IBM Tivoli Management Framework through the Tivoli Desktop

Figure 4-5: Installing the fix pack

Figure 4-6: Error message

Figure 4-7: Specifying the path to the installation image

Figure 4-8: Install Patch

Figure 4-9: Patch Installation

Figure 4-10: Filling out the Add a Serial Network to the HACMP Cluster SMIT screen

Figure 4-11: HACMP for AIX SMIT screen

Figure 4-12: Configure nodes to an HACMP Cluster

Figure 4-13: Enter service IP label for tivaix1

Figure 4-14: Enter service IP labels for tivaix2

Figure 4-15: Configure Resources to Make Highly Available SMIT screen

Figure 4-16: Configure Application Servers SMIT screen

Figure 4-17: Fill out the Add Application Server SMIT screen for application server tws_svr1

Figure 4-18: Fill out the Add Application Server SMIT screen for application server tws_svr2

Figure 4-19: How to select an application server to monitor

Figure 4-20: Add Process Application Monitor SMIT screen for application server tws_svr1

Figure 4-21: COMMAND STATUS SMIT screen after creating HACMP process application monitor

Figure 4-22: Add Process Application Monitor SMIT screen for application server tws_svr2

Figure 4-23: Normal operation of two-node mutual takeover HACMP cluster

Figure 4-24: Location of application servers after tivaix2 falls over to tivaix1

Figure 4-25: State of cluster after tivaix2 returns to service and reintegrates with the cluster

Figure 4-26: Running jobs under TWS Engine2 on tivaix1 prevent TWS Engine2 from transferring back to tivaix2

Figure 4-27: TWS Engine2 on tivaix1 is quiesced, only held jobs exist on tivaix1 under TWS Engine2. TWS Engine2 can now fall back to tivaix2

Figure 4-28: File /etc/hosts copied to all by cluster nodes of cluster we used

Figure 4-29: Enter service IP label for tivaix1

Figure 4-30: How to enter service IP labels for tivaix2

Figure 4-31: Add an IP-Based Network to the HACMP Cluster SMIT screen

Figure 4-32: Configure resource group rg1

Figure 4-33: How to configure resource group rg2

Figure 4-34: Define resource group rg1

Figure 4-35: Define resource group rg2

Figure 4-36: Set cascading without fallback (CWOF) for a resource group

Figure 4-37: Add a Custom Cluster Event SMIT screen

Figure 4-38: Add quiesce_tws script in Change/Show Cluster Events SMIT screen

Figure 4-39: Select a Node SMIT dialog

Figure 4-40: Add a Persistent Node IP Label/Address SMIT screen for tivaix1

Figure 4-41: Network Name SMIT dialog

Figure 4-42: Add a Persistent Node IP Label/Address SMIT screen for tivaix2

Figure 4-43: Select Add a Pre-defined Communication Interface to HACMP Cluster configuration

Figure 4-44: Select the Pre-Defined Communication type SMIT selector screen

Figure 4-45: Select a Network SMIT selector screen

Figure 4-46: Add a Communication Interface SMIT screen

Figure 4-47: COMMAND STATUS SMIT screen for successful verification of an HACMP Cluster configuration

Figure 4-48: COMMAND STATUS SMIT screen for our environment's configuration

Figure 4-49: Start Cluster Services SMIT screen

Figure 4-50: COMMAND STATUS SMIT screen displaying successful start of cluster services

Figure 4-51: Current status of all HACMP subsystems on a cluster node

Figure 4-52: Select a Resource Group SMIT dialog

Figure 4-53: Select a Destination Node SMIT dialog

Figure 4-54: Move a Resource Group SMIT screen

Figure 4-55: COMMAND STATUS SMIT screen for moving a resource group

Figure 4-56: How to start HACMP on system restart

Figure 4-57: Relationship of IBM Tivoli Workload Scheduler, IBM Tivoli Management Framework, Connectors, and Job Scheduling Consoles during normal operation of an HACMP Cluster

Figure 4-58: Viewing multiple instances of IBM Tivoli Workload Scheduler on separate cluster nodes on a single display

Figure 4-59: Available scheduling engines when logged into tivaix1 during normal operation

Figure 4-60: Available scheduling engines when logged into tivaix2 during normal operation

Figure 4-61: How multiple instances of the Connector work during normal operation

Figure 4-62: Multiple instances of Connectors after tivaix2 falls over to tivaix1

Figure 4-63: Sample error dialog box in Job Scheduling Console indicating possible fallover of cluster node

Figure 4-64: Available scheduling engines on tivaix1 after tivaix2 falls over to it

Figure 4-65: Available Connectors in interconnected Framework environment after tivaix2 falls over to tivaix1

Figure 4-66: Available Connectors in interconnected Framework environment during normal cluster operation

Figure 4-67: IBM Tivoli Framework 4.1.0 application and patch sequence and dependencies as of December 2, 2003

Figure 4-68: Available scheduling engines after interconnection of Framework servers

Figure 4-69: Log into TWS Engine1

Figure 4-70: Log into TWS Engine2

Figure 4-71: Network diagram of the Microsoft Cluster

Figure 4-72: Cluster Administrator

Figure 4-73: Installation-Select Language

Figure 4-74: Installation-Welcome Information

Figure 4-75: Installation-License agreement

Figure 4-76: Installation-Install new Tivoli Workload Scheduler

Figure 4-77: Installation user information

Figure 4-78: Installation install directory

Figure 4-79: Type of Installation

Figure 4-80: Type of IBM Tivoli Workload Scheduler workstation

Figure 4-81: Workstation information

Figure 4-82: Extra optional features

Figure 4-83: Installation of Additional Languages

Figure 4-84: Review the installation

Figure 4-85: IBM Tivoli Workload Scheduler Installation progress window

Figure 4-86: Completion of a successful install

Figure 4-87: Cluster Administrator

Figure 4-88: IBM Tivoli Workload Scheduler Workstation definition

Figure 4-89: New Cluster resource

Figure 4-90: Resource values

Figure 4-91: Node selection for resource

Figure 4-92: Dependencies for this resource

Figure 4-93: Resource parameters

Figure 4-94: Registry Replication

Figure 4-95: Cluster resource created successfully

Figure 4-96: Dependencies for IBM Tivoli Workload Scheduler Netman service

Figure 4-97: Cluster Administrator

Figure 4-98: The Advanced tab

Figure 4-99: General tab for Group Properties

Figure 4-100: Failover tab for Group Properties

Figure 4-101: Failover tab for Group Properties

Figure 4-102: Network diagram of the Microsoft Cluster

Figure 4-103: IBM Tivoli Workload Scheduler user authentication flow

Figure 4-104: Installation location for TMRs

Chapter 5: Implement IBM Tivoli Management Framework in a Cluster

Figure 5-1: IBM Tivoli Management Framework in normal operation on tivaix1

Figure 5-2: State of cluster after IBM Tivoli Management Framework falls over to tivaix2

Figure 5-3: Start SSA link verification on tivaix1

Figure 5-4: Results of link verification test on SSA adapter ssa0 in tivaix1

Figure 5-5: Results of SSA link verification test on SSA adapter ssa0 in tivaix2

Figure 5-6: Select an SSA disk from the SSA Physical Disk SMIT selection screen

Figure 5-7: Identify the connection address of an SSA disk

Figure 5-8: Select physical volumes for volume group itmf_vg

Figure 5-9: Configure settings to add volume group itmf_vg

Figure 5-10: Select a volume group using the Volume Group Name SMIT selection screen

Figure 5-11: Create a standard Journaled File System on volume group itmf_vg in tivaix1

Figure 5-12: Successful creation of JFS file system /opt/hativoli on tivaix1

Figure 5-13: Rename a logical volume

Figure 5-14: REname the logical log volume

Figure 5-15: Export a Volume Group SMIT screen

Figure 5-16: Import a volume group

Figure 5-17: Import volume group itmf_vg on tivaix2

Figure 5-18: IBM Tivoli Framework 4.1.0 application and patch sequence and dependencies as of December 2, 2003

Figure 5-19: Normal operation of highly available Endpoint

Figure 5-20: Fallover operation of highly available Endpoint

Figure 5-21: Normal operation of local and highly available Endpoints

Figure 5-22: Cluster state after moving highly available Endpoint to tivaix2

Figure 5-23: Cluster state after falling over tivaix1 to tivaix2

Figure 5-24: Configure Resources to Make Highly Available SMIT screen

Figure 5-25: Configure Application Servers SMIT screen

Figure 5-26: Fill out the Add Application Server SMIT screen for application server itmf

Figure 5-27: Select an application server to monitor

Figure 5-28: Add Process Application Monitor SMIT screen for application server itmf

Figure 5-29: COMMAND STATUS SMIT screen after creating HACMP process application monitor

Figure 5-30: Configure resource group rg1

Figure 5-31: Define resource group rg1

Figure 5-32: Set cascading without fallback (CWOF) for a resource group

Figure 5-33: Start the installation using setup.exe

Figure 5-34: Framework [advanced] installation screen

Figure 5-35: Tivoli License Agreement

Figure 5-36: Accounts and file permissions screen

Figure 5-37: Framework hostname configuration

Figure 5-38: Framework installation password

Figure 5-39: Tivoli Remote Access Account (TRAA) setup

Figure 5-40: Framework installation type

Figure 5-41: Framework license key setup

Figure 5-42: Framework setting review

Figure 5-43: Tivoli oserv service installation window

Figure 5-44: Framework installation completion

Figure 5-45: File copy output

Figure 5-46: Create the etc\tivoli directory on node 2

Figure 5-47: Copy the Tivoli environment files

Figure 5-48: Source the Tivoli environment

Figure 5-49: Add the Tivoli account

Figure 5-50: Copy the tivoliap.dll

Figure 5-51: Register the tivoliap.dll

Figure 5-52: Installing Autotrace

Figure 5-53: Create the oserv service

Figure 5-54: Create a new resource

Figure 5-55: Resource name and type setup

Figure 5-56: Configure possible resource owners

Figure 5-57: TRIP dependencies

Figure 5-58: TRIP service name

Figure 5-59: Registry replication

Figure 5-60: Create a new resource

Figure 5-61: Resource name and type setup

Figure 5-62: Select owners of the resource

Figure 5-63: Select resource dependencies

Figure 5-64: Service and parameter setup

Figure 5-65: Registry replication

Figure 5-66: Bringing cluster resources online

Figure 5-67: Tivoli setup

Figure 5-68: Start TRIP installation

Figure 5-69: TRIP Welcome screen

Figure 5-70: The TRIP license agreement

Figure 5-71: Installation directory configuration

Figure 5-72: Installation confirmation

Figure 5-73: TRIP installation screen

Figure 5-74: TRIP installation completion

Figure 5-75: ManagedNode installation

Figure 5-76: Add Clients dialog

Figure 5-77: Tivoli Framework installation media

Figure 5-78: Tivoli Framework installation options

Figure 5-79: Specify a Tivoli access account

Figure 5-80: Client installation screen

Figure 5-81: Tivoli Remote Access Account (TRAA) setup

Figure 5-82: Managed Node reboot screen

Figure 5-83: Managed Node installation window

Figure 5-84: Starting the oserv service

Figure 5-85: Configure Managed Node IP address

Figure 5-86: Endpoint overview

Figure 5-87: Endpoint advanced configuration

Figure 5-88: Endpoint login verification

Figure 5-89: Sample last.cfg file

Figure 5-90: Registering the lcfd service

Figure 5-91: lcfd and lcfd-2 services in the registry

Figure 5-92: Endpoint login verification

Figure 5-93: Add a new cluster resource

Figure 5-94: Name and resource type configuration

Figure 5-95: Possible Owners

Figure 5-96: Dependency configuration

Figure 5-97: Add lcfd-2 service name

Figure 5-98: Registry replication

Figure 5-99: Completion dialog

Figure 5-100: Bring resource group online

Appendix A: A Real-Life Implementation

Figure A-1: Our environment

Figure A-2: IBM Tivoli Workload Scheduler HACMP configuration flowchart

Figure A-3: Add a TTY

Figure A-4: Our environment in more detail

Figure A-5: IPAT via aliasing topology example

---