Creating a Network Cabling Infrastructure

Imagine that you have a bunch of data center devices and they need to be connected to each other. You could connect them using individual cables for every connection. Chapter 4, "Determining Data Center Capacities" described a set of racks, 40 Sun Fire 6800 servers with 4 Sun StorEdge racks connected to each Sun Fire server. The RLU definition for a Sun StorEdge rack contained 8 multi-mode fibre connections (40 x 4 x 8, or 1,280 multi-mode fibre cables). This makes 1,280 fibre cables running under the floor just to support disk connectivity to these configurations. Let's say you also want to manage the Sun StorEdge T3 arrays in these racks over your network. You need another 1,280 Cat5 cables to just the Sun StorEdge racks, plus 40 Cat5 cables to connect the Sun Fire 6800 servers to your administration network, plus 40 cables to connect the Sun Fire 6800 servers to only one of your production networks, plus 40 cables to connect the consoles of these devices. That's 2,680 separate cables running under the floor going to different devices in different locations on the floor. In an ideal world, each Sun Fire 6800 server and its 4 Sun StorEdge racks would be right next to each other. However, they probably aren't, so you have cables criss- crossing under the floor.

Now, suppose one of those cables goes bad and you need to replace it. If it's not labeled, you need to physically trace the cable under the floor. The probability is that the cable you need to trace is wrapped up in a bunch of other cables and will be difficult and time-consuming to trace, ensure that it is the correct cable, and replace. There is a better way to solve this problem. By knowing your connectivity requirements you can create a modular design using points of distribution (PODs) which minimize unnecessary cabling under the floor.

Determining Connectivity Requirements

Make sure you read Chapter 4, "Determining Data Center Capacities," and determine your connectivity requirements for each device.

The connectivity requirements will be based on the type of connections the device has (Cat5 or fibre) and how many of these connections you need for each device. For example, a Sun StorEdge T3 array has one fibre connection and two Cat5 connections, one for network connection and one for the physical console. You need the fibre connection to transfer data to and from the Sun StorEdge T3 array. To configure, administer, and monitor the Sun StorEdge T3 array through the network, you need a connection to the network port through its Cat5 interface. If you want access to the physical console as well, this is again through a Cat5 cable. Let's say you want network connectivity but not the physical console. For each Sun StorEdge T3 array you need one multi-mode fibre cable and one Cat5 cable. With eight Sun StorEdge T3 arrays in a rack, the connectivity requirement is eight multi-mode fibre and eight Cat5.

Modular Design

In the past, when the cabling requirements for machines were less (maybe one or two per machine), you could run the cables to one central point, usually the network room. However, as you can see from the previous example, the number of connections has increased by orders of magnitude. You can still run 2,680 cables back to the network room, but the data center design philosophy dictates that you keep the design as simple as possible.

Since we have segmented the floor into a given number of RLUs of particular types, we can define an area on the floor that contains a certain number of RLUs which will determine how many Cat5 and fibre connections the area will need. Repeat this process for all areas of the floor. Each of these clusters of RLUs, and more specifically , their network cabling requirements, can be looked at as a module. This also allows us to build in some fudge factor. It is as likely as not that, over time, some RLUs will be over their initial cabling requirements and others will be below. By grouping some of them together we have the flexibility (another part of the design philosophy) to allocate an extra connection from an RLU that is not in use to one that needs it. We can also locate support devices, switches, terminal servers, and Cat5 and fibre patch panels for this module somewhere within this cluster of RLUs.

You might need to connect a storage device on one side of the data center to a server on the opposite side. There are two ways to do this. You can use the logic contained in switches to move data from one device to another, or you can use the patch panels to cross-connect one patch panel port to another. This basic design allows you to keep connections local to an area for greater simplicity, but gives you the flexibility to connect (logically or physically) from one module to another.

Hierarchy of the Network Structure

The previously described design relies on the fundamental structure of logical networking that allows you to create a hierarchy of devices. The following figure shows an example hierarchy of devices.

Figure 9-1. Hierarchy of Network Devices

Consider this example of a very simple network hierarchy structure. The machine Cranmer needs to send data to a machine called Wolsey. Cranmer sends the data to Sub-Switch A. Sub-Switch A does not have a direct connection to Wolsey so it sends out a request to ask if any device with a direct connection can send the data to Wolsey. The Master Switch responds, so Sub-Switch A sends the data to the Master Switch. The Master Switch sends the data to Sub-Switch B which is directly connected to Wolsey and sends the data to Wolsey. This usually happens within a few hundredths of a millisecond. You could extend this hierarchy to thousands of machines and tens or hundreds of switches, 20 levels deep. The time required to get the data through will increase with the increased number of levels in the hierarchy (this is called latency), but the basic operation is the same.