17.3 Quorum and Voting

Quorum is a simple majority. A cluster will not form without quorum. If a cluster is running and loses too many votes, then quorum is lost, and the cluster will suspend.

When a cluster is suspended, all access to cluster managed storage, all process activity, and any network operations are suspended. The only exception is that the CNX and DRD subsystems continue to function so that they can:

Quiesce the cluster.
Drain any outstanding I/O in process.
Continue to access the cnx partition on the cluster member and quorum disks.
Attempt to reform the cluster from surviving members.

Note

the cluster may continue to function for a period of time after quorum is lost; however, all application I/O directed at cluster storage will be suspended to preserve data integrity.

The cluster will appear hung, but it's not hung. If you check the console, you will see output similar to the following:

 CNX MGR: quorum lost, suspending cluster operations.

The Connection Manager continues to function, attempting to accumulate enough votes to regain quorum so that the cluster can resume operation.

 CNX MGR: quorum (re)gained, (re)starting cluster operations.

Every system that is added to the cluster can be a voting member or a non-voting member. Furthermore, a quorum disk^[4] can be added to the cluster to increase availability and act as a tiebreaker in the event of a cluster partition. Currently, a system can have one vote or zero votes (the same is true for the quorum disk).

Every member (or potential member) determines what quorum is based on the "Quorum Algorithm."

17.3.1 The Quorum Algorithm

The CNX calculates the number of votes needed for quorum based on the number of votes it expects to see in the cluster if all voting members (and the quorum disk if configured and assigned a vote) are up.

The first thing that the CNX needs to do is determine the maximum number of booting nodes that are fully connected (meaning that each node can communicate with every other node).

Secondly, the CNX will determine the Cluster's Expected Votes (CEV) from the set of "fully connected" nodes.

The CEV is defined as the maximum number of votes from the following:

The previous CEV.

The previous CEV only applies if the cluster is going through a membership transition and not when the cluster is first forming (as the CEV from a previous cluster incarnation is not stored).
The highest member-specific cluster_expected_votes attribute value.

This is the clubase:cluster_expected_votes attributes from each member's sysconfigtab file in the /.local../boot_partition/etc directory.

Normally this value should be the same for all members.
The sum of node votes (cluster_node_votes) and quorum disk votes (cluster_qdisk_votes).

The cluster_node_votes and cluster_qdisk_votes are located in the clubase subsystem. The specific values are defined in each member's sysconfigtab file in the /.local../boot_partition/etc directory.

Finally, the CNX will determine the number of votes required to reach quorum.

The quorum votes (QV) value is calculated as follows:

             QV = round_down ((CEV + 2)/2)

Here's another way of looking at this, especially if you're a programmer:

             QV = (CEV >> 1) + 1

For example, in a three-member cluster where each member has one vote, every member should have a cluster_expected_votes value of 3.

If the CEV is 3:

QV = (3 + 2) / 2
QV = 5 / 2
QV = 2.5
QV = 2

So in order to form a cluster, at least two systems must be booted.

Here's a little one line Perl script to calculate quorum.

             # perl -e 'printf("%d\n", (CEV >> 1) + 1);'

 # perl -e 'printf("%d\n", (3 >> 1) + 1);' 2

Let's consider that the cluster is fully up, and we add a fourth member using the clu_add_member(8) command. The "BEFORE" values are what the CNX will be using prior to the clu_add_member command. The "AFTER" values are what the CNX will be using after the clu_add_member command but before booting the newly added member. And the "BOOT" values are what the CNX will use after the new member has been booted and attempts to join the cluster.

	BEFORE	AFTER	BOOT
CEV	3	3	4
QV	2	2	3
Current Votes	3	3	4
Member-Specific `cluster_expected_votes`	3	3	4

The cluster's previous expected votes value was 3. However, when the new member is added, each member's expected votes value will be modified in their sysconfigtab file to 4. Since 3 members are up, the current number of votes is 3.

Once the new member is booted, the CEV will be recalculated to 4, which would increase the QV to 3.

If QV = 3, then 3 out of 4 cluster members would have to be booted in order for a cluster to form. At this point adding a quorum disk to the cluster would increase the cluster's availability.

	BEFORE	AFTER
CEV	4	5
QV	3	3
Current Votes	4	5
Member-Specific `cluster_expected_votes`	4	5

By adding the quorum disk, the QV value does not change, but the number of potential votes has increased by 1 thus decreasing the number of members required to reach quorum.

Table 17-4 shows the number of votes required to form a cluster as well as recommendations on when to configure in a quorum disk.

Table 17-4: Votes Required for Quorum
Votes Required For Quorum QV = (CEV >> 1) + 1
# of Members with 1 Vote	Quorum Disk with 1 Vote	Expected Votes (CEV)	Votes Required for Quorum(QV)	% of Voting Members Required for Quorum	Recommendation
2	x	2	2	100%	Bad Idea
2	√	3	2	67%	Highly Recommended
3	x	3	2	67%	Highly Recommended
3	√	4	3	75%	Bad Idea
4	x	4	3	75%	Bad Idea
4	√	5	3	60%	Highly Recommended
5	x	5	3	60%	Highly Recommended
5	√	6	4	67%	Bad Idea
6	x	6	4	67%	Okay
6	√	7	4	57%	Recommended
7	x	7	4	57%	Highly Recommended
7	√	8	5	63%	Bad Idea
8	x	8	5	63%	Okay
8	√	9	5	56%	Recommended

^[4]For more information on the quorum disk, see section 17.8.3.