10.4 Configuration management

I get very frustrated when I have a group of systems that are all seemingly configured the same way and one of those systems exhibits a particular problem that none and others seem to experience. The problem exhibited may be performance related or be as severe as frequent system crashes. In my travels and discussions with Exchange system managers, I can’t tell you how often I hear stories of system problems that, upon further diagnosis, are caused by dissimilarities in configuration. There are literally thousands of configurable parameters throughout Windows Server alone. Exchange Server adds a further dimension to this. A deviation from the standard of one parameter on a particular server can wreak havoc on a deployment and may go undiscovered until it causes severe problems. In addition, configuration parameters are not the only cause of configuration-management issues. In fact, configuration-management problems are usually software or hardware related. Table 10.4 lists the most common software and hardware causes of configuration-management problems that I have commonly seen in Exchange deployments.

Table 10.4: Common Software and Hardware Configuration-Management Challenges

Windows service packs	Service packs for Windows Server that include bug fixes and enhancements are released on a regular basis, and it is a constant challenge to keep current across an entire deployment. Furthermore, service packs require a reboot of the operating system and are sometimes challenging to schedule.
Application service packs (Exchange Server)	Exchange Server service packs are also released regularly and have similar challenges to operating system service packs.
Third-party application updates and service packs	Invariably, most Exchange servers must run some amount of third-party software, such as management applications and agents, backup agents, and antivirus software. Ensuring homogeneity among third-party software can also be a challenge.
Hardware device drivers	Most hardware vendors have specific device drivers for devices such as network interface cards and disk controllers.
Hardware firmware (ROMs)	Most hardware vendors have firmware that is frequently updated to address issues. Within a server, the system, disk controller, and other devices, such as tape drives and disk drives, all contain firmware that must be maintained with current versions.

Solid configuration-management practices are key to a mission-critical Exchange deployment. You will need to establish practices and guidelines for maintaining the highest degree of homogeneity across your Exchange deployment. Once you develop and establish the degree and means of enforcement, you will need to determine which tools you will employ to monitor, troubleshoot, and resolve configuration-management issues. Tools in Windows such as the Registry Editor, Windows Diagnostics, Hardware Device Manager, and Resource Kit utilities will assist in configuration management without additional investments. However, I also recommend that you invest in a third-party management application that has strong configuration management capabilities for both the operating system and Exchange Server. Look specifically for applications with change-management or control features. Tools such as Microsoft Systems Management Server (SMS), NETIQ’s AppManager, or BMC’s Patrol provide excellent configuration-management features.

Figure 10.3: HP’s Insight Manager version control feature.

Don’t forget about your hardware vendor in all of this. Most top-tier server vendors provide excellent server-management tools such as HP’s Compaq Insight Manager (CIM). CIM (shown in Figure 10.3) provides some of the best configuration management available in terms of the hardware, device driver, and firmware version control.

The best method to get a handle on configuration management across your Exchange deployment is the practice of blockpointing . Blockpoints are manageable units of change that you can roll out to your entire server population on a regular basis. Blockpoints are packages that are tested and certified for deployment. Blockpoints may include all of the components listed in Table 10.4 or just a subset. In practice, your system operations and deployment staff set a schedule for blockpoints for your deployment.

Typically, I have seen organizations set specific periods for blockpoints such as quarterly or semiannually. One popular timing method for blockpoints is to tie them to operating system service pack updates, although the irregular schedule of Windows SP releases can make this a challenge. Once you have determined a blockpoint schedule, you will need to set up a process for blockpoint qualification and certification. This usually involves methods for determining what components will be part of blockpoints and the process in which they will be tested and certified for deployment. The process typically involves gathering all required components and testing them together in a lab environment that mirrors your deployment environment. Here you will determine any incompatibilities or “gotchas” and devise methods for how the blockpoint will be applied.

The final step will be to schedule the blockpoint process for your deployment. For large deployments, it may not be possible to apply the blockpoint across the entire deployment over a weekend. The time required, as well as the impact on users, will need to be considered. You should also determine a course of action (a backout plan) in the event that a blockpoint fails and renders a server unavailable. It is better to think about this in advance (i.e., proactively) and have your fallback measures readily available. The blockpoint approach to configuration management can truly be a lifesaver that maximizes reliability. I have seen many instances in the past several years of Exchange Server outages that could have been avoided if configuration management monitoring and practices such as blockpointing had been in place. If you do not have measures similar to blockpoints in place, consider adding this practice to your operations portfolio.