Power Distribution System Design

A well-designed electrical system for the data center ensures adequate and consistent power to the computer hardware and reduces the risk of failures at every point in the system. The system should include dedicated electrical distribution panels and enough redundancy to guarantee constant uptime. A well-designed electrical system will provide consistent power and minimize unscheduled outages. Equipment subjected to frequent power interruptions and fluctuations is susceptible to a higher component failure rate than equipment connected to a stable power source.

Electrical work and installations must comply with local, state, and national electrical codes.

Assessing Power Requirements

Usually, your electrical design firm will tell you how much power is coming into the building as DC (Direct Current) which is expressed by KVA (Kilo Volt Amps). The easiest way to express this is in watts. When using DC power, volts x amps = watts (VxA=W). For example, you might be told that there is 7500KVA and that 7000KVA is available to the data center. The other 500KVA is needed for the rest of the building for offices, copiers, lighting, smoke detectors, soda machines, etc.

You can use the rack location units (RLUs) you've determined for your design to calculate how much power you need for equipment. The RLU definitions should include not only servers and storage equipment, but also network equipment such as switches, routers, and terminal servers. Add to this the power requirements of your HVAC units, fire control systems, monitoring systems, card access readers, and overhead lighting systems.

From your RLU definitions, you know that you'll need 800 30Amp 208V L6-30R outlets to power all of your racks. However, most circuit breakers will trip when they reach 80 percent of their rated capacity (this is sometimes referred to as a 0.8 diversity factor). A 30Amp breaker will really only allow a maximum of 24Amps through it before it trips and shuts down the circuit. Each circuit can handle about 5000 watts (24 amps x 208 volts = 4992 Watts) or 5KVA so the worst case electrical draw per outlet is 5KVA x 800 outlets = 4000KVA. No problem, because this is well within the 7000KVA you have allocated. However, most of the watts that these racks consume go into producing heat, and it will take quite a bit more electricity (for HVAC) to remove that heat.

A good rule of thumb is to take your total equipment power and add 70 percent for the HVAC system. The electrical usage will vary depending on the system and climatic conditions. Your HVAC and electrical designers should be able to give you a more precise multiplier once the HVAC system specifics are known.

4000KVA x 1.7 = 6800KVA, and that is within the 7000KVA you have been allocated. So, now you know that you have a large enough power in-feed to meet your electrical requirements.

The previous example uses the maximum draw that the breaker can accommodate before it trips. Most racks will not draw the full 5KVA, and it is possible that they could draw considerably less. The example of watt usage for RLU-A in Chapter 4, "Determining Data Center Capacities" is 3611 watts. This works out to a diversity factor of .58 (30Amps x 208 volts x .58 = 3619.2 watts). If you are building a data center that will be filled with just RLU-A racks, you could use a .58 diversity factor. However, this would mean that your average watts per RLU could not exceed 3619 watts. If you need to use a diversity factor below .80, you should use the highest wattage definition from all of your RLUs to determine the diversity factor. Also you must consider that power requirements will go up over time, so adding in an extra 3 to 5 percent to the diversity factor will also provide some headroom for next generation products that you can't anticipate during the design stages.

Finally, consider all possible future modifications, upgrades, and changes in power needs. For example, installing 50Amp wiring when only 30Amp is currently needed might be worth the extra cost if it is likely, within a few years , that machines will be added that need 40 to 50Amp wiring. The initial cost could be insignificant compared to the cost of dismantling part of the data center to lay new wire.

Consider the following questions during the design process:

• Is a certain amount of power allocated for the data center?

• Will power sources be shared with areas outside the data center?

• Where will the power feeds come from?

  • Will redundant power (different circuits or grids) be available?

  • Historically, how often do multiple grids fail simultaneously ?

  • If power availability or dependability is a problem, can a power generating plant be built?

• Will the data center need single-phase or three-phase power (or both)?

• If the existing site is wired with single-phase, can it be retrofitted for three-phase?

• If you intend to use single-phase, will you eventually need to upgrade to three-phase?

  • Can you use three-phase wire for single-phase outlets, then change circuit breakers and outlets later when three-phase is needed?

• Where will the transformers and power panels be located? Is there a separate space or room for this?

• Which RLUs and their quantities need two independent power sources for redundancy?

• Will there be UPS? Where will the equipment be located?

• If there is only one external power feed, can half the power go to a UPS?

• Can a UPS be connected only to mission-critical equipment?

Multiple Utility Feeds

The availability profile of the data center could be the determining factor in calculating power redundancy. Ideally, multiple utility feeds should be provided from separate substations or power grids to ensure constant system uptime. However, those designing the center must determine whether the added cost of this redundancy is necessary for the role of the data center. It will be related to the cost of downtime and whatever other power delivery precautions you are taking. If you have a requirement for your own power generation as backup for data center power, then the additional costs of multiple utility feeds might not be cost effective. You should get historical data from your power supplier on the durations of outages in your area. This can be valuable information when making these decisions.

Uninterruptible Power Supply

An Uninterruptible Power Supply (UPS) is a critical component of a highly-available data center. In the event that power from the grid should fail, the UPS should be able to power 100 percent of the hardware for at least the amount of time needed to transfer power from an alternative utility feed or from backup generators. It should also be able to carry 150 percent of the power load to accommodate fault overload conditions. Don't forget to factor in the minimum HVAC power requirements. Also, include the power requirements needed for emergency lighting and any electronic equipment needed to access the data center, such as access card readers.

Figure 7-1. Control Panel and Digital Display of a UPS System

You might size your UPS to accommodate the actual power draw rather than the total power draw. For example, a machine might use 1500 watts for "normal" load. However, when it's powered on, it might initially draw 2200 watts. This load of 2200 watts is the "peak" load. You should size the UPS to handle this peak load.

However, this means a larger and more costly UPS. If budget is an issue, you will be taking a risk if you use a UPS rated for your normal load as it might fail to meet the peak load.

The UPS should be continually online, used to filter, condition, and regulate the power. Battery backup should be capable of maintaining the critical load of the room for a minimum of 15 minutes during a power failure to allow for the transfer of power from the alternate source, or to bring machines down cleanly if an alternate power source is not available. If a UPS is not used, surge suppression should be designed into the panels and a stand-alone isolation/regulation transformer should be designed into the power system to control the incoming power and protect the equipment.

Backup Power Generators

Backup power generators should be able to carry the load of the computer equipment, as well as all support equipment such as HVACs and network equipment. Depending on the availability status of the data center, it might be acceptable to use the UPS and multiple utility feeds without generators. If, by researching the power supply history, you determine that outages of 15 minutes or less are likely, you should install a UPS system with 20 minutes of battery power. This will sustain the data center until power is back on. If there is an outage of longer than 20 minutes, the data center will go down. This decision must be based on risk exposure determinations. The probability of a 20-minute outage might not outweigh the cost of generators.

If you plan for the use of generators, you'll need to think about code compliance, where they will be located (they give off exhaust), where the fuel tanks will be placed (one company used the same size tank used in gas stations , and it had to be buried), whether or not additional concrete pads must be installed, etc. You must also consider contracts with diesel suppliers.

Sharing Breakers

Though it is sometimes a necessary evil, sharing breakers is not recommended. As described in the earlier sections, machines don't use all of the capacity of their resident circuits. You have a normal load and a peak load. Two machines, each with a normal load of 1500 watts and a peak load of 2200 watts, could share the same 5KVA 30Amp circuit. However, if the configuration of these devices is changed over time, for example, if more memory is added, this might change the normal and peak loads, over the amount that the circuit could handle. While you might be forced to do this, you must be very careful and accurate in your power usage calculations for any circuit that you share.

Figure 7-2. Breaker Panel

Maintenance Bypass

The power system design should provide the means for bypassing and isolating any point of the system to allow for maintenance, repair, or modification without disrupting data center operations. The system should be designed to avoid all single points of failure.

Installation and Placement

The power distribution equipment for computer applications should be installed as close as possible to the load. All loads being supported must be identified and evaluated for compatibility with the computer equipment. Heavy loads that are cyclic, such as elevators, air conditioners, and large copy machines, should not be connected directly to the same source as the data center equipment.