High-Availability Cluster Design Goals

To be highly available with no single point of failure, our cluster must handle these situations in the following ways:

• If the Primary Director no longer responds to requests for cluster resources on its public network interface, the secondary Director should take over and shut off or reset the power to the Primary Director (to avoid a split-brain condition).

• If one of the real servers (cluster nodes) inside the cluster no longer responds to requests for its services, the real server should be removed from the cluster.

• If the Director is operating in LocalNode^[1] mode and a locally running daemon stops responding, the Director should stop routing requests for cluster resources locally and forward all requests to real servers inside the cluster instead.^[2]

A highly available cluster should also:

• Use separate physical switches for the primary and secondary Director in order to avoid a single point of failure with the network hardware.

• Use a highly available NAS server in order to avoid a single point of failure with the shared file system. (See Chapter 16 for a discussion of a shared storage technique that address lock arbitration problems.)^[3]

  Note 

  Many LVS clusters will not need to use NAS for shared storage. See Chapter 20 for a discussion of SQL database servers and a brief overview of the Zope project—two examples of sharing data that do not use NAS.

• Use a hardened Linux distribution such as EnGarde or LIDS (http://www.lids.org) in order to protect the real servers when the real servers are connected to the Internet.

Note 

See "LVS defense strategies against DoS attack" on the LVS website for ways to drop inbound requests in order to avoid overwhelming an LVS Director when it is under attack. Building a cluster that uses these techniques is outside the scope of this book.

^[1]LocalNode was introduced in Chapter 12.

^[2]Although this is normally a very unlikely scenario.

^[3]Your NAS vendor will have a redundancy solution for this problem as well.