8.4 A history lesson in Exchange Server clustering

When Microsoft set out to provide clustering support in Exchange Server 5.5, the development team had two primary, closely related goals. The first was to increase the availability of Exchange Server by adding clustering support. The second was to provide protection from hardware failures.

Distributed processing, load balancing, and data backup were not part of these original goals. As a result, when released in late 1997 (shortly after MSCS in Windows NT Server Enterprise Edition), Exchange Server 5.5 provided only rudimentary fail over capabilities when paired with MSCS. Exchange Server 5.5 Enterprise Edition implemented support for MSCS using the Service Fail over mode described earlier.

One generic resource DLL was used for each Exchange service (i.e., Information Store, Directory Service, Message Transfer Agent, and System Attendant). This allows individual services to be brought down without causing complete fail over, and it also isolates services, which helps ease troubleshooting. Functionality supported in Exchange Server 5.5 in a clustered environment can be classified into two categories: supported services and unsupported services. Supported services include those required to provide basic Exchange Server functionality such as the Information Store, Directory Service, Message Transfer Agent, and System Attendant. In addition, the Internet Mail Service and the Internet News Service support clustered functionality (with the exception of the dial-up mode of these services). These supported services provide automatic startup on fail over, automatic recovery processing, and global registry updates when moving between cluster nodes. The remaining Exchange Server 5.5 functionality falls into the unsupported category.

Cluster support in Exchange 2000

While, in the opinion of many, Exchange 5.5 clustering support was abysmal, it was only the first step. The design goal for Exchange 2000 Server was to build on the initial clustering support provided in Exchange 5.5 Enterprise Edition and to provide full application functionality in a clustered environment. For one, Exchange 2000 Server offered initial support for Active/Active clustering not offered in Exchange 5.5. When any member resource of a resource group fails on a cluster node, the resource group will be failed over to another node in the cluster that will take over the services being provided by that resource group. The goal for Exchange 2000 was to support one or more Exchange 2000 virtual servers in the cluster, so that each virtual server would run on one of the nodes in the cluster. Unfortunately, while this support was designed into Exchange 2000, Microsoft and its customers learned that such configurations were problematic and subsequently discouraged rich and high-scale Active/Active cluster configurations in Exchange 2000 SP1.

From an administrative perspective, all Exchange components required to provide services and a unit of fail over are grouped (via a cluster resource group) into an Exchange virtual server (EVS) in Exchange 2000. An EVS, at a minimum, will include a storage group and required protocols, plus any required cluster resources such as an IP address, network name, or physical disk. From the viewpoint of MSCS, an EVS exists as a collection of resources in each cluster resource group. Clients connect to the virtual servers the same way they would connect to a stand-alone server. The cluster service monitors the virtual servers in the cluster. In the event of a failure, the cluster service restarts or moves the affected virtual servers to a healthy node. For planned outages, the administrator can manually move the virtual servers to other nodes. In either event, the client will see an interruption of service only during the brief time that the virtual server is in an online/off-line pending state.

Supported Exchange 2000 components

Clustering support differs according to the component. Not all components of Exchange 2000 are supported in a clustered environment. These components are the resources that makeup the resource group for an Exchange 2000 virtual server. Table 8.4 details which components are supported and, in some cases, the type of clustering they are capable of supporting.

Table 8.4: Exchange 2000 Cluster-Supported Services

Component	Operational Mode	Remarks
Exchange SMTP Service	Active/Active	Depends on the Information Store Service. Implemented as SMTP virtual server.
Exchange IMAP4 Service	Active/Active	Depends on the Information Store Service. Implemented as IMAP virtual server.
Exchange POP3 Service	Active/Active	Depends on the Information Store Service. Implemented as POP3 virtual server.
Exchange HTTP (OWA) Service	Active/Active	Depends on the Information Store Service. Implemented as HTTP virtual server.
Exchange Information Store Service	Active/Active	Depends on the System Attendant Service. Implemented as Information Store virtual server.
Exchange Routing Service	Active/Active	Depends on the System Attendant Service. Implemented as Routing Service virtual server.
Exchange Message Transfer Agent (MTA)	Active/Passive	Can only be one MTA virtual server per cluster (created on the first Exchange virtual server configured). All Cluster Exchange virtual servers are dependent on this MTA virtual server for legacy Exchange message routing support. Depends on the System Attendant Service.
Exchange MS Search Service	Active/Active	Provides content indexing function and is dependent on Information Store Service. Implemented as MS Search virtual server.
Exchange System Attendant Service	Active/Active	Fundamental resource that manages the creation of all other Exchange virtual server resources. Dependent on a network name and disk(s) resource.

Exchange 2000 components not supported on MSCS include:

• Microsoft Active Directory Connector (ADC)

• Exchange Chat Service

• Exchange Conference Services

• Exchange Instant Messaging

• Exchange Key Management Service (KMS)

• Exchange Calendar Connector

• Exchange Messaging Connectors including: Lotus cc:Mail, Notes/ Domino, Microsoft Mail, Novell GroupWise

• Exchange Event Service

• Exchange Site Replication Service

• Exchange NNTP Service

Exchange 2000 provides core features and support for Microsoft Cluster Service via two key components, as shown in Figure 8.5. These are the Exchange cluster administration DLL and the Exchange resource DLL.

Key component: Exchange resource DLL

If you recall the earlier discussion of cluster-aware versus cluster-unaware applications, you’ll remember that it is the existence of an application-specific resource DLL that is the key differentiator for cluster-aware applications. Also remember that Exchange 5.5 did not provide its own resource DLL, but made use of the generic resource DLL provided with MSCS. For Exchange 2000, Microsoft developers took the extra time and effort to guarantee full cluster functionality. The result of this effort is the Exchange resource DLL called EXRES.DLL. This DLL for Exchange 2000 is installed when the setup application realizes that it is operating in a clustered environment. EXRES.DLL acts as a direct resource monitor interface between the cluster services and Exchange 2000 by implementing the Microsoft Cluster Service API set. As you may recall, Cluster Service uses EXRES.DLL to communicate with Exchange and manage its components. EXRES.DLL allows Exchange resources to be managed (brought on-line and off-line), proactively checks resource states via IsAlive API calls (for more information about cluster resource DLLs and the LooksAlive and IsAlive calls, see the Microsoft Developer Network (MSDN) at http://msdn.microsoft.com), reports resource failures to the Resource monitor, and performs other resource-management functions for Exchange components in the cluster.

Key component: Exchange cluster administration DLL

In Figure 8.5, you notice that Exchange also interacts with the cluster management API that is implemented as part of the Cluster Administrator (CLUSADMIN.EXE). In order for Exchange resources to be configured and controlled by the Cluster Administrator, they must enable Exchange Services to communicate with the Cluster Administrator and for the Cluster Administrator program to provide Exchange-specific configuration parameters and screens. The Exchange cluster administration DLL (EXCLUADM.DLL), provides this support. The Exchange cluster administration DLL provides the necessary wizard screens when configuring Exchange resources in the Cluster Administrator and presents Exchange resources that can be added as resources in the cluster such as the Microsoft Exchange System Attendant. The cluster administration DLL is a key component in the configuration and management of Exchange services in the cluster. It is not required for resource monitoring and restart or fail over actions. The Exchange resource DLL (EXRES.DLL) performs this role.

Figure 8.5: The Exchange Cluster Administration DLL and the Exchange resource DLL.

Exchange 2000 cluster resource dependencies

Figure 8.6 illustrates a tree structure of Exchange resource dependencies. Exchange services must have certain resources as predecessors before they can be brought on-line as cluster resources. By default, Exchange 2000 installs nine resources in the form of virtual servers into a cluster resource group that is being configured in the Cluster Administrator.

When configuring cluster resources for Exchange, four prerequisites must be satisfied. First, Exchange 2000 must be installed on the cluster nodes where Exchange resources will run. Next, an IP address and network name must be created for the Exchange virtual server being configured. Since the network name resource is dependent on the IP address, the IP address must be created first. The final step before configuring Exchange specific resources is to allocate the physical disk resources required by the virtual server you are configuring. At a minimum, there must be at least one physical disk resource configured for Exchange virtual servers to be added to the cluster configuration. When Exchange cluster resources start and stop (change states), they must do so in order of resource dependency. This means that, on startup, resources start in forward order of resource dependence (bottom to top in Figure 8.6). When resources are stopped (or a resource group is taken off-line), resources are shut down in reverse order of dependence (top to bottom in Figure 8.6). When configuring cluster resources, having a firm grasp on the resource dependencies for Exchange clustering makes the task much simpler.

Figure 8.6: Exchange 2000 Cluster resource dependency tree.

Since the virtual server is the unit of fail over, Exchange 2000 virtual server resources will all fail over in the cluster as a managed unit. This means that the entire cluster resource group containing the virtual server resources will be failed over together from one node in the cluster to another. One or more Exchange information storage groups can be configured as part of an Exchange 2000 virtual server. However, a storage group can only belong to one virtual server. The virtual server is the key unit of client access, cluster management, and fail over for Exchange 2000 services running in a clustered environment. When deploying Exchange 2000 clusters, ensure that you familiarize yourself with virtual servers and how they are used.

8.4.1 What has changed in Exchange Server 2003 clustering?

While I know you are hopeful that, based on all you have heard, things are much better with Exchange Server 2003 cluster, the truth is that not much has changed. What has changed is Microsoft’s understanding of the best practices and supportable Exchange cluster configurations. Microsoft learned a lot from its mistakes around Exchange 2000 clustering support. When Exchange 2000 shipped Microsoft and customers had high hopes for Exchange clusters. However, Microsoft did not invest as much as it should have. Exchange 2000 cluster configurations went largely untested prior to the release of Exchange 2000. By the time Microsoft and customers realized there were problems, it was too late and everyone was left with a somewhat low opinion of Exchange 2000 clustering.

While this is unfortunate, Exchange 2003 has benefited from this little misadventure. Microsoft and customers have spent a lot of time investigating the issues with Exchange clustering and Microsoft has made two key advances. First, Microsoft has done a huge amount of testing and now better understands how to make Exchange cluster deployments successful. There is a mountain of knowledge yielding prescriptive deployment guidance (take a look on the Exchange Web site and Microsoft TechNet) and other documentation for Exchange 2000 clusters than has been developed in the almost 3-year period that Exchange 2000 has been available. Second, Microsoft has made every effort and attempt to fix cluster bugs and design flaws within the current Exchange architecture. This all means that while some key issues that impact the success of Exchange cluster deployments cannot be addressed within current code base of Exchange 2000/2003, Microsoft has done everything possible to overhaul Exchange 2003 clustering and provide customers with everything they need to be successful with Exchange clusters.

Cluster memory management

When Exchange 2000 was released, customers and Microsoft immediately learned some lessons about Exchange 2000 clustering. One of the most problematic issues was memory management on Exchange 2000 clusters. Specifically, virtual memory was soon recognized as a scarce commodity for Exchange 2000 clusters. Since Windows implements a virtual memory model (shown in Figure 8.7) that is based on a linear 32-bit address space, 4 GB of virtual memory is available from an architectural viewpoint. However, Windows allocates (by default) 2 GB of this to applications and 2 GB to the operating system. It is possible to increase the application-addressable virtual memory by using the /3GB BOOT.INI switch, but in some cases, this is still not sufficient.

From a practical point of view, Exchange 2000 clusters had issues with virtual memory allocation because of the fact that the STORE.EXE process is a huge consumer of virtual memory. While Exchange does implement Dynamic Buffer Allocation (DBA), this mechanism was not sufficient to address the issues encountered with Exchange 2000 clusters. The problem that Exchange 2000 administrators encountered with clusters was that clusters under heavy load or with large configurations could potentially run out of virtual memory. This could potentially cause server failure or prevent cluster fail over. The core issue relates to the fact that, over time, virtual memory (VM) in Windows becomes fragmented as applications and the operating system allocate and deallocate blocks. When an Exchange 2000 cluster was under load or heavily configured (such as with a full complement of storage groups and databases), the potential exists for severe VM fragmentation and the absence of enough free blocks to support Exchange 2000 cluster operations (e.g., allocation and virtual server fail over). The end result was unstable or unreliable clusters. While Microsoft acted quickly to alleviate many of these issues in Exchange 2000 SP1–SP3 (the major improvements that Microsoft added to Exchange 2000 in SP 1 through Exchange Server 2003 RTM are highlighted in Table 8.5), the damage was done and many lost confidence in Exchange 2000 clustering.

Figure 8.7: Windows virtual memory model.

I prefer not to provide redundant content on the subject of Exchange clustering. Therefore, for more information on Exchange 2000 cluster deployment, I highly recommend the Microsoft white paper entitled: “Deploying Exchange 2000 Server Clusters” at www.microsoft.com/ downloads/release.asp? ReleaseID=44277. Information and best practices in this paper apply to both Exchange 2000 and 2003.

For Exchange Server 2003, Microsoft put a significant amount of work into addressing operational issues with clusters. First, virtual memory usage was a focus of a significant amount of work and bug fixes. Exchange Server 2003 is a better application in terms of the ways it uses and frees virtual memory on Windows. This results in less VM fragmentation and greater confidence in Exchange clusters. Microsoft also looked at how passive cluster nodes operated. In Exchange 2000, the passive node continued to run Exchange services (such as the information store—STORE.EXE) and would consume virtual memory. This created the potential to impede fail over to the passive node since virtual memory allocations could potentially fail in heavy load or configuration scenarios. For example, if a passive node had been running for an extended period of time, there may a lower amount of free and nonfragmented virtual memory available. In the event of a cluster node failure, an Exchange virtual server being failed over may not have adequate VM available to fail over—resulting in an error during fail over. In Exchange Server 2003, when an Exchange virtual server is either moved or fails over to another node, the information store process is stopped, thereby freeing all virtual memory allocated to STORE.EXE. This cluster node is then prepared to accept a virtual server and restart the information store process with almost guaranteed availability of a free VM block for use by Exchange. The memory-management improvements made since Exchange 2000 release have drastically improved the viability of Exchange clusters. Exchange Server 2003 adds to these improvements, but doesn’t get Exchange administrators off the hook. Even with these improvements, it is still crucial to monitor virtual memory performance on Exchange 2003 clusters (more on this later).

Improving cluster fail over

The resource dependency model from Exchange 2000 cluster resources (shown in Figure 8.8) provided adequate management of Exchange resources in the cluster environment. However, the nature and depth of this dependency tree also complicated Exchange 2000 cluster fail overs. As discussed earlier, the resources must be taken off-line and on-line in dependency order. For example, in a cluster, the Exchange protocol resources (IMAP, POP3, HTTP, SMTP) cannot be brought on-line until the Information store resource is in an on-line state. This tiered dependency model adds additional time to Exchange virtual server fail over and startup (cluster resources going between on-line and off-line states). While it is impossible to totally alleviate this problem, Microsoft made a key improvement to address this situation. The dependency hierarchy of the Exchange resources has been flattened (as shown in Figure 8.8) to decrease the amount of time that it takes to bring Exchange virtual servers on-line and off-line (thereby improving fail over). This improvement has the effect of allowing Information Store databases (supported by the information store cluster resource) and protocol services to come on-line in parallel when a fail over occurs. Other resources also benefit from this reduced degree of dependency. By combining this flattened resource-dependency model with the use of Windows Cluster Service’s antiaffinity API (which allows the cluster service to better determine fail over nodes and resource conditions and node affinities), Exchange virtual servers can fail over faster. Exchange virtual server fail over speed may still be limited by other factors such as log file replay, but these improvements will help. Ultimately, these improvements should benefit Exchange 2003 cluster deployments by decreasing virtual server fail over times and cluster stability and, thereby, reducing downtime (by minutes per fail over).

Figure 8.8: Exchange Server 2003 Cluster resource dependency tree.

Managing Exchange clusters better

One of the key issues in the initial release was not simply that fact that there were issues like virtual memory fragmentation existed, but that it was difficult for Exchange administrators to manage. Microsoft simply did not provide adequate monitoring and alerting for Cluster fail over failures and VM fragmentation issues. Table 8.5 highlights the key improvements in Exchange Server monitoring and memory management since the release of Exchange 2000, including those in Exchange Server 2003. Using these improvements Exchange administrators now have the tools they need to better manage Exchange Server clusters—ensuring that cluster configurations serve their intended purpose: to increase availability, not reduce it.

Table 8.5: Exchange Cluster Monitoring and Memory-Management Improvements

Improvement	Description
Exchange Virtual Memory Utilization Event Logging	Microsoft added Exchange virtual memory resource checking and event logging to ensure that administrators were alerted when low fragmented virtual memory conditions occur on Exchange servers and clusters. Event ID 9582—Warning Condition: VM fragmentation resulting in performance degradation Recommendation: Schedule a restart of Exchange services Event ID 9582—Error Condition: VM fragmentation resulting in failures Recommendation: Restart Exchange services immediately
Exchange Virtual Memory Performance Monitoring	Microsoft added four new counters to the MSExchangeIS performance object. VM largest block size: Largest free block of VM VM total 16-MB free blocks: The number of free VM blocks ≥ 16 MB VM total free blocks: The total number of free VM blocks VM total large free block bytes: The sum of total of all blocks ≥16 MB
Top-Down Memory Allocation Scheme	Changed the way the STORE.EXE process allocates memory to a scheme that is more conducive to reducing VM fragmentation issues.

Another important manageability issue with Exchange 2000 clusters was that the cluster permissions model created challenges with the installation, setup, configuration, and management of clusters. The permissions to create, delete, or modify resources or the Exchange virtual server were updated in Exchange 2003.

For Exchange 2000 clusters, the permissions model requires the cluster service to enforce a rather challenging permission scheme, depending on whether the Exchange virtual server is the first in the Exchange organization or not, in order for an administrator to be able to create, delete, or modify the Exchange virtual server resources. This permissions model (shown in Table 8.6) presented some challenges for large enterprises with distributed Exchange administration models. For example, if an Exchange administrator wants to create an Exchange virtual server in a decentralized site where administration is delegated to a lower level (the administrator has Full Admin at the Administrative Group level, but not at the Organizational level), he or she must either be granted temporary Full Admin at the Organizational level or have someone with those permissions manage the Exchange virtual server for him or her. In addition, the Windows Cluster Service needed to have Exchange Full Administrator permissions at the administrative group level.

Table 8.6: Exchange 2000 Cluster Permissions Model

Exchange Virtual Server Status	Permission/Administrator Role Required
First in the organization	Exchange Full Administrator (Organization level)
Second or later in the organization	Exchange Full Administrator (Administrative Group level)

Microsoft updated this model for Exchange Server 2003 to simplify administration for Exchange clusters and to alleviate issues like the one I discussed above. The specific permissions to create, delete, or modify Exchange virtual servers depends on the mode your Exchange organization is running—mixed (Exchange 2000/Exchange 2003) or native mode (Pure Exchange 2003)—and the topology (note that in this case, mixed mode refers to a Exchange 2000/Exchange 2003 environment, not Exchange 5.5/Exchange 2000). The four following key points summarize the changes to the Exchange Server 2003 cluster permissions model:

1. To install Exchange 2003 on a cluster node, you must have local administrator privileges on that node (this also makes you a cluster administrator).

2. To create the first Exchange virtual server in the organization, you still need Exchange Full Administrator permissions at the Organizational level (unfortunately, this issue was not easily resolved in the current architecture).

3. To create subsequent Exchange virtual servers in the organization, you must have Exchange Full Administrator permission at the Administrative Group level (again, this issue was not addressed).

4. The Cluster Administrator account no longer needs Exchange Full Administrator privileges at the Administrative Group level.

Overall, the real key change made was to the cluster administrator requirements and an improvement in what’s required to install Exchange 2003 on a cluster node. The “First Virtual Server” issue still remains, as it was not something Microsoft could address in the current cluster support architecture for the Exchange 2000/2003 code base.

Exchange resource monitoring improvements

Windows cluster service interaction with cluster resources mainly depends on the resource DLL and its implementation of the IsAlive and LooksAlive API calls. The implementation of IsAlive and LooksAlive in Exchange 2000 versus Exchange 2003 is not vastly different. However, minor subtleties can make a difference. For Exchange 2000, Microsoft simply “instrumented” EXRES.DLL to behave with the same logic for both API calls (properly implemented, LooksAlive is a notification call pushed from the resource DLL, whereas, IsAlive is a poll-driven call to the resource DLL from the cluster resource monitor). Regardless of which was used, the response from EXRES.DLL was the same. For Exchange 2003, some minor enhancements were made to improve Exchange 2003’s reliability and stability in a cluster. The interaction of Exchange 2003’s EXRES.DLL with the cluster resource monitor is illustrated in Figures 8.5 and 8.9. For LooksAlive (the less invasive, but more frequent API call), the resource monitor can poll the EXRES.DLL instead of having to wait for it to report its status (which is how it worked in Exchange 2000). The more invasive, but less frequent IsAlive call is implemented in a similar manner in EXRES.DLL for Exchange 2003 as it was in Exchange 2000. IsAlive simply allows the cluster resource monitor to look at a cached memory location where EXRES.DLL has reported its state (this interaction and timeouts for IsAlive calls can be configured as properties on a per-resource basis from the Cluster Adminstrator).

Figure 8.9: Cluster service interaction with Exchange 2003 via IsAlive and LooksAlive.

Exchange 2003 cluster supported components

As Table 8.7 reveals, the list of Exchange 2003–supported components has not changed since Exchange 2000.

Exchange 2003 did not bring earth-shattering improvements to Exchange clustering (versus those offered in Exchange 2000 SP1-SP3).

Table 8.7: Exchange 2003 Cluster Supported Services

Component	Operational Mode	Remarks
Exchange SMTP Service	Active/Active	Depends on the Information Store Service. Implemented as SMTP virtual server.
Exchange IMAP4 Service	Active/Active	Depends on the Information Store Service. Implemented as IMAP virtual server.
Exchange POP3 Service	Active/Active	Depends on the Information Store Service. Implemented as POP3 virtual server.
Exchange HTTP (OWA) Service	Active/Active	Depends on the Information Store Service. Implemented as HTTP virtual server.
Exchange Information Store Service	Active/Active	Depends on the System Attendant Service. Implemented as Information Store virtual server.
Exchange Routing Service	Active/Active	Depends on the System Attendant Service. Implemented as Routing Service virtual server.
Exchange Message Transfer Agent (MTA)	Active/Passive	Can only be one MTA virtual server per cluster (created on the first Exchange virtual server configured). All Cluster Exchange virtual servers are dependent on this MTA virtual server for legacy Exchange message routing support. Depends on the System Attendant Service.
Exchange MS Search Service	Active/Active	Provides content indexing function and is dependent on Information Store Service. Implemented as MS Search virtual server.
Exchange System Attendant Service	Active/Active	Fundamental resource that manages the creation of all other Exchange virtual server resources. Dependent on a network name and disk(s) resource.

Many may argue that Exchange 2003 simply completed the work that Microsoft should have done for Exchange 2000. Regardless, Microsoft has managed to invest a significant amount of effort in making clustering more robust for Exchange 2003. This was done by identifying the key destabilizing factors for Exchange 2000 clusters (such as fail over, resource dependency, and virtual memory allocation) and attempting to address as many issues as possible within the boundaries of the current architecture limitations. Microsoft also invested a lot of effort into testing more Exchange 2003 cluster configurations and providing more extensive design and deployment guidance than ever before. In my opinion, Exchange 2003 clusters are a viable solution to the problem and challenge of building mission-critical Exchange servers.