Chapter 2: The Basics of Achieving High Availability

In every high availability solution, as alluded to in Chapter 1, Preparing for High Availability, you must adhere to some fundamentals; otherwise , achieving the availability you need will be difficult. Even if you are in an IT shop or department that takes care of many of the basics listed in this chapter, you should still find a few nuggets of information here or you might be bored. Either way, it would be inappropriate for a book on high availability to ignore the fundamentals.

This chapter addresses matters that relate to both the human and technological factors of the high availability equation. It includes such topics as data center best practices, staffing, service level agreements (SLAs), and change management.

Data Center Best Practices

This section starts with a few funny ”and yet not so funny ”stories that are not only based on real events, but more common than you would think.

• A company was experiencing an availability problem intermittently. At approximately 2 A.M. a few nights a month, the main accounting database server went down and then automatically came back about an hour later. In some cases, at least one of the databases running was marked suspect in Enterprise Manager, causing it to be rebuilt. One occurrence would be an isolated incident, but this was clearly more than that. The CIO, CFO, and CTO launched a joint investigation. In the end, the culprit was not a denial of service attack, it was a janitor attack: the janitor was unplugging the system to plug in cleaning equipment.

• SQL Server was experiencing intermittent failures with no apparent root cause or predictable timing. As in the first example, a joint investigation was launched to determine what was going on. The problem turned out to be that telephone maintenance workers were walking past the cabling, which was loose and tangled, shifting cable plugs a bit. To add insult to injury , in the main wiring room that contained the core networking and telephony wires, the telephone workers occasionally touched some of the network wiring in addition to the telephony ones, causing some network outages.

• A junior database administrator (DBA), who was new to the company and did not know the systems, their purpose, or everything about how the company administered servers, started a backup process. The DBA did something incorrectly ”such as backing up to the wrong disk volume and filling up a disk subsystem critical for another SQL server ”and caused a serious availability problem until the situation was corrected.

Have any of these events, or something similar, happened where you currently work or have worked in the past? In most companies, at least, the first example is preventable by using qualified, professional, and trained cleaning staff. The others might not be as easily solved , especially the third one if, say, all the other DBAs are unavailable and the junior DBA is left at the helm. Proper training would not cover the improper backup situation, but it would eliminate the other problems (not knowing the systems, and so on).

Chances are, you inherently know the best practices that relate to your production environment ”many books and sessions at conferences talk about them. But how many do you adhere to, and how many are barriers to your availability? Each topic in this section could be more detailed, but the ideas that follow will serve as guidelines.

As a database system engineer or administrator, you might not have complete authority over how your SQL Server is eventually hosted or managed, but you should be able to observe and document discrepancies you see. If problems persist, take your documentation, along with records of downtimes, to management to influence and change how the servers and systems you are responsible for are hosted and managed.

Even if you do not run the data center, have no direct involvement or access, or have no input on its potential construction, it is important to let the proper people know about the things listed next . Whether you are using a third-party hosting company or keeping everything in-house, consider location; security; cabling, power, communication systems, and networks; third-party hosting; support agreements; and the under the desk syndrome.

Location

The location of your data center will contribute to its availability. Do not locate or use a data center that is under plumbing lines, sewer lines, or anything similar ” basically anything that could cause problems. What would happen if a pipe burst, sending water through the ceiling of the data center and flooding the server room? Think about what businesses are above and below your offices. If, for example, your offices are located under a kitchen, not only is the potential for fire increased, but also leaks could seep through the kitchen floor. Is it wise to put your data center near something that could go up in smoke at any minute?

If none of this can be avoided, consider the options listed throughout this section and see if they help mitigate risk to the data center. If they do not, or if some of them are not feasible due to high costs or other reasons, the risk will have to be documented and taken into account. Some situations are unavoidable, even in the best of IT shops .

• Proper climate control of systems is crucial for availability. There is a big difference between hosting one server and hosting 1000 servers in the same location. Computers do not work well if they are overheating due to the climate, which might also cause anomalies such as condensation if the conditions in the room are just right. Make sure you have a properly installed heating, ventilation , and air conditioning (HVAC) system ”with the emphasis on properly installed. This means that there will be sufficient air circulation to keep the entire room at a uniform, acceptable temperature. Conversely, if your data center is in an extremely cold geographic climate, you might also need a heating system because systems do not work well in extreme cold, either.

• Make sure that the entire data center s airflow is even, otherwise there could be hot and cold spots. If certain areas are too cold, there s the risk of an administrator adjusting the air conditioning if he or she is cold while working in the room and raising the temperature in areas other than their own, causing system outages in the now-hot areas. Maintaining systems at the proper temperature extends their mean time between failures (MTBF) and the life of the hardware; in extreme temperatures some hardware might not function at all.

• Electrical lines must be properly grounded. Failing to have this done can potentially result in some bizarre behaviors in electrical equipment. Newer buildings should, for the most part, be grounded, but some older buildings might not be. Computer components , especially small ones like CPUs and memory, do not react well when sudden power surges occur. Power conditioners might be needed to guarantee that there is a consistent, even amount of power flowing to all things electric.

• Servers should be neatly and logically organized in racks (see Figure 2-1) where all equipment can be reached. The location of each server should be documented so that in the event of an emergency, an affected server can be easily located. Racks also help protect the equipment from damage, protect against most accidental situations (such as a person accidentally knocking a wire loose), and foster air circulation to assist with proper climate control.

  
  Figure 2-1: A good example of servers arranged neatly in racks.

• To enable the equipment to be rolled into the data center on racks, you might need specific clearance for both height and width; otherwise, you might not be able to get the equipment into the data center. Conversely, if you need to move the equipment out, you must account for that as well (for example, if you have custom racks built into the data center instead of brought in). Also, make sure that ramps can hold the weight of the equipment being rolled in and out, as well as accommodate people in wheelchairs. The ramps should not be too narrow.

• Plan for growth on many levels. Much like capacity planning for your systems themselves , invest the time to assess how much physical space is currently needed and how much will be needed over the next few years. What is the planned life of the data center? One specific example of something small and seemingly insignificant is making sure there are enough electrical outlets in the room to allow for additional equipment over the years . In no case do you want to have to resort to power strips to expand the power, risking overload and outages. You must also take into account the increased power consumption and its affect on such things as the HVAC.

• A fire suppression system should be installed in the data center. If at all possible, make sure that it will not further damage the computer systems and endanger human life. It s imperative that anyone who works in or near the server room reacts properly to a fire alarm and adheres to all building rules, including taking part in fire drills. If the fire alarm cannot be heard , there should be some other visual indicator in the data center to alert the staff. One concern about some fire systems is that they are just as dangerous to human beings as they are to computer systems. Halon depletes the ozone layer, and some fire suppression systems can turn artificial ceiling materials into a deadly weapon.

• Make sure that there is enough room above and below the server racks to allow the cables and climate control system to be installed properly (see Figure 2-2). Also, make sure the racks will fit into the room ”do not prepurchase racks only to find out that they will not fit your room, as well as allow for the clearance and airflow needed.

  
  Figure 2-2: The proper amount of clearance above the racks as well as room for the cabling.

• Climate and geographic location are important factors in the placement of a data center. If you work in a place susceptible to volcanic eruptions, earthquakes, hurricanes, tsunamis, or other natural disasters, take that into account in your planning. Because of these limitations imposed by nature ”you cannot stop a hurricane from flooding or, in a worst case scenario, destroying your data center ” you might need to consider a second location in a separate geographical area to act as a secondary data center, or cold site.