STORAGE AREA NETWORKS

One solution to the storage dearth is implementing a SAN. SANs are networks that are designed, built, and maintained with one purpose in mind: to store and transfer data. SANs are a burgeoning field, with impressive growth expected.

Storage Needs

With the popularity of the Internet and the massive increase in e-commerce, organizations are scrambling for a means to store vast amounts of data. A popular way to maintain terabytes of information employs SANs, which interconnects storage devices with Fibre Channel switches. Even though data storage is cheap-you can add a 200-GB hard drive for a couple hundred dollars-this is a reactive response to storage shortages. By creating a patchwork of hard drives, your internetwork's overhead escalates, and you slowly lose control.

SANs, on the other hand, allow you to manage all your storage needs in a proactive manner while maintaining the high availability that you need. Figure 10-1 shows an example of how a LAN and a SAN work together.

Figure 10-1: SANs and LANs operate independently, but are still able to mesh together

As organizations and their computing needs grow, so will their reliance on data storage. For instance, as more and more companies add server farms to manage their internal and external affairs, the more reliable the internetwork must be. To ensure high availability, servers sharing storage pools in a SAN can failover with no hiccup in service. Furthermore, because fiber optics are the backbone of SANs, disaster recovery is pared from several hours to a few minutes or less.

By combining LAN networking models with the core building blocks of server performance and mass storage capacity, SANs eliminate the bandwidth bottlenecks and scalability limitations imposed by previous SCSI bus-based architectures. In addition to the fundamental connectivity benefits of SAN, the new capabilities, facilitated by SAN's networking approach, enhance its value as a long-term infrastructure. These capabilities, which include clustering, topological flexibility, fault tolerance, high availability, and remote management, further elevate a SAN's ability to address the growing challenges of data-intensive, mission-critical applications.

There are three primary components of a storage area network:

• Interface The interface is what allows storage to be external from the server and allows server clustering. SCSI, Fibre Channel, iSCSI, and Fibre Channel over IP (FCIP) are common SAN interfaces.

• Interconnect The interconnect is the mechanism these multiple devices use to exchange data. Devices such as hubs, routers, gateways, and switches are used to link various interfaces to SAN fabrics.

• Fabric The platform (the combination of network protocol and network topology) based on switched SCSI, switched fiber, and so forth. The use of gateways allows the SAN to be extended across WANs.

Fibre Channel

Fibre Channel is an industry-standard, high-speed serial interface for connecting PCs and storage systems. Fibre Channel provides attachment of servers and storage systems across distances up to 100 km (which is about 4,000 times farther than parallel SCSI interfaces). This allows the storage facilities to be located on another floor, another building, or in another city.

Fibre Channel carries 200 times more bandwidth than SCSI (4 Gbps versus 20 Mbps). In addition, Fibre Channel supports multiple standard protocols (like TCP/IP and SCSI) concurrently over the same physical cable. This is useful because it simplifies cabling and keeps down infrastructure costs. Because Fibre Channel allows standard SCSI packets to be transported across fiber-optic lines, existing SCSI devices can be maintained and used alongside Fibre Channel devices.

Note 

Although Fibre Channel is most often used with fiber-optic connections, it can still be used with copper wiring. However, there are more speed and distance limitations imposed on copper deployments than on fiber. In addition, copper can suffer performance degradation due to electromagnetic interference.

For the sake of reliability through redundancy, Fibre Channel SANs should be built around SAN hubs and switches. This ensures that no single point of failure exists and that performance bottlenecks are ameliorated. This should sound familiar, because it is a key consideration when designing a LAN.

Fibre Channel Layers

The Fibre Channel standard contains five layers. Each layer is responsible for a specific set of functions. If you think back to Chapter 2, you might notice some commonalities between Fibre Channel and the OSI model. In the Fibre Channel model, the layers are numbered FC-0 through FC-4.

Table 10-1 describes the different layers of the Fibre Channel model, and Figure 10-2 illustrates this stack.

Table 10-1: Layers in the Fibre Channel Stack

Layer	Description
FC-0: Physical Layer	Defines cabling, connectors, and the signaling controlling the data. This layer is akin to the OSI physical layer.
FC-1: Transmission Protocol Layer	Responsible for error detection, link maintenance, and data synchronization.
FC-2: Framing and Signaling Protocol Layer	Responsible for segmentation and reassembly of data packets that are sent and received by the device. Additionally, sequencing and flow control are performed at this layer.
FC-3: Common Services Layer	Provides services like multicasting and striping.
FC-4: Upper Layer Protocol Mapping Layer	Provides the communication point between upper layer protocols (like SCSI) and the lower Fibre Channel layers. This layer makes it possible for more than SCSI data to traverse a Fibre Channel link.

Figure 10-2: The Fibre Channel Stack contains five layers

Although the stacks are different (not the least of which is that the OSI model has seven layers, while Fibre Channel has five), each layer in the model relies on the layer immediately above or below it.

As Fibre Channel utilizes the layer format, products and applications performing at one layer are compatible with products and applications residing at another layer, which is what the OSI model does.

Going the Distance

SCSI storage solutions have to be located next to the server because its range is limited to 25 meters. This proved to be troublesome, especially if space in a server room was at a premium. Because Fibre Channel allows for long-distance locations between servers and storage devices, the two pieces of equipment can be up to 100 km apart.

Three technologies can be added to Cisco SAN devices to provide Fibre Channel connectivity. These technologies employ different laser wavelengths to garner varying levels of bandwidth and distance. Those wavelengths are:

• Short wavelength (SWL) Providing connectivity up to 500 meters

• Long wavelength (LWL) Providing connectivity up to 10 km

• Coarse Wavelength Division Multiplexing (CWDM) Providing connectivity up to 100 km

In practice, however, the ability for data to travel long distances can be useful if a storage center is constructed to maintain all the data for several departments. Furthermore, several servers located at one campus could send their data to a central storage facility in a separate building. This allows for the creation of modular and scalable storage pools.

Increased Connectivity

Using Fibre Channel also simplifies the connectivity of multiple systems accessing a shared storage device by overcoming the limitations of parallel SCSI, including distance and number of devices per bus. Fibre Channel supports eight times more devices per loop than parallel SCSI. In practice, however, it may not be realistic to put so many devices on a single loop. However, the capability now exists for large numbers of servers to access storage devices like RAID arrays or tape libraries.

Other SAN Protocols

In addition to Fibre Channel, there are a couple of other SAN protocols to be aware of. These protocols are useful in designing and deploying the optimal SAN, and can be used as part of your SAN solution. Let's talk about two of the currently prevalent protocols (FCIP and iSCSI) and rub our crystal ball for a look at the future of SAN protocol iFCP.

iSCSI

The first protocol is the basket in which Cisco seemed to put all of its eggs a few years ago-Small Computer System Interface over IP. In essence, this is simply transported data in the SCSI protocol across TCP/IP networks. SCSI is the language of disk drives, and iSCSI is a protocol that encapsulates SCSI commands and data for transport across IP networks. It is beneficial, because it interoperates with existing applications and operating systems, as well as with LANs and WANs.

Products that use iSCSI allow hosts on an IP network to connect across Gigabit Ethernet networks to Fibre Channel or SCSI storage. IP storage networks are built directly on top of existing IP networks. It didn't take off like Cisco had originally hoped, but don't read this to mean that iSCSI is a dead technology. It is still a viable protocol and used in many of Cisco's (and other vendors') SAN devices. In fact, iSCSI and Fibre Channel are both prevalent in SAN deployments.

FCIP

A newest protocol is FCIP, or Fibre Channel over IP. FCIP represents two distinct technologies (storage networking and long-distance networking) merged together. FCIP combines the best attributes of both Fibre Channel and the Internet Protocol to connect distributed SANs. FCIP encapsulates Fibre Channel and sends it over a TCP socket.

FCIP is considered a tunneling protocol because it makes a transparent point-to-point connection between geographically disparate SANs, utilizing IP networks. FCIP relies on TCP/IP services for connectivity between SANs over LANs, MANs, and WANs. TCP/IP is also tasked with congestion control and management, as well as with data error and data loss recovery. The benefit of all this is that organizations can leverage their existing technology investments by extending the Fibre Channel fabric over an IP link.

Veltech10V3.indd 415 Veltech10V3.indd 415 11/4/06 9:11:11 PM

iFCP

Just to confuse matters with FCIP, another SAN protocol is iFCP. This stands for Internet Fibre Channel Protocol. Even though the letters are the same (though in a different order to really confuse things), the technology is rather different. iFCP allows an organization to extend Fibre Channel across the Internet using TCP/IP. This sounds a lot like FCIP, but that is where the similarities end. While FCIP is used to extend a Fibre Channel fabric with an IP-based tunnel, iFCP is a movement away from current Fibre Channel SANs toward the future of IP SANs.

iFCP gateways can complement existing Fibre Channel fabrics or completely supplant them. iFCP allows organizations to create an IP SAN fabric, minimizing the Fibre Channel component and maximizing the use of TCP/IP infrastructure.

While Cisco's solution, which we'll talk about in the next section, is based largely on FCIP, it is helpful to know what is on the horizon with iFCP.

Designing and Building a SAN

When it comes down to designing and building a SAN, it's necessary to consider several important factors before plugging fiber into routers and switches. You should consider such issues as what kind of applications you'll be using, the best design for the backbone, how you'll configure your topology, and what mechanisms you'll use to manage your SAN. Let's take a closer look at each of these issues.

Application Needs

When developing and designing a SAN, the first step is to figure out which applications will be served. No matter if you're designing a common data pool for a bank of Web servers, a high-performance data-streaming network, or any other needs, you must pay special attention to the SAN infrastructure. You have to take into consideration such issues as port densities, distance and bandwidth requirements, and segmentation. These are all variables that are affected by the application.

Note 

In a mixed environment, it's important to evaluate the platforms that will compose the SAN. Hardware and software support for SANs varies, depending on which platforms you use. Once you have addressed these fundamental questions, you can begin constructing the SAN.

A SAN's construction is similar to a typical Ethernet infrastructure. A SAN comprises a few basic components: the Fibre Channel disk storage and tape libraries, fiber hubs and switches, host bus adapters (HBA), and some form of SAN management.

Backbone

As you design your SAN, a critical architectural hardware decision is whether to use arbitrated loop or switched fabric:

• Arbitrated loop Shares bandwidth and employs round-robin data forwarding. At one time, it was the only choice for SAN backbones.

• Switched fabric Dedicates full bandwidth on each port and allows simultaneous data transfers to a single node.

Your choice will be decided based largely on your scaling and performance needs. If you have modest storage needs, a simple hub should be enough to get the job done. On the other end of the spectrum, larger storage environments almost demand fiber switches.

In small groups or SOHOs (small office/home office), a good foundation is a Fibre Channel hub in an arbitrated-loop configuration. Hubs are well suited for this environment because they provide a high level of interoperability for a reasonably low price. Hubs support an aggregate bandwidth of 100 Mbps. Hubs can support up to 127 de vices, but for optimal results, you should limit it to about 30 devices. Furthermore, because the per-port costs of a switch are higher than those of a hub, a hub is best to fan out the core switch ports to the connecting servers.

One of the main reasons hubs are limited in their scalability is because of the way devices are added into the loop. In order to recognize other devices in the loop, each loop must perform a loop initialization sequencer ( LIP ) when it is first attached to the network. When this action is performed, the loop is suspended while the entire membership on the loop acquires or verifies the port addresses and is assigned an arbitrated loop physical address. Although the recognition process is quite fast, time-sensitive traffic (like VoIP or data backups) can be negatively affected by these performance speed bumps.

On the other hand, hubs are useful because they are inexpensive, easy to configure, and interoperate well with other hubs and other vendors' products.

Fiber

If SANs are so fantastic, why are we just hearing about them now? The main factor that has brought SANs into play as a viable technology is the use of switched fiber.

Fiber switches support 4-Gbps full-duplex on all ports. Unlike hubs, which, as we mentioned earlier, require an LIP, a fiber switch requires nodes connected to its ports to perform a fabric logon. The switch is the only device that sees this logon, and this allows devices to enter and exit the fabric without providing an interruption to the remaining devices.

Devices on an arbitrated-loop hub, which is cascaded off a switch, are not fundamentally compatible with other devices on the fabric. Unlike a switched LAN environment, devices in a switched SAN environment must perform a fabric logon to communicate with other devices. However, those devices that are not built with fabric support usually cannot operate over fabric, because they don't perform a fabric logon. Rather, they use an LIP.

Configuration

Just as in a LAN, there are several ways to configure switches, providing different levels of performance and redundancy. In a SAN, the basic configuration design is the treetype model. In this scheme, switches cascade off one another and fan out throughout the SAN, shown in Figure 10-3.

Figure 10-3: The tree model is the basic topology for a SAN

The main problem with this model is its scalability constraints due to the latency inherent with the single-port interface. This also limits bandwidth and is not ideal because it can be a single point of failure. This type of design is best as an alternative to fiber hubs for a SAN that has just a single-tier cascade.

For larger, more intricate SANs, the best choice for both high availability and performance is the mesh model. The mesh makes a large network of switches: Each switch is connected to every other switch, thereby eliminating the opportunity for a single point of failure. The mesh also reduces bottlenecks and latency. The mesh model is illustrated in Figure 10-4.

Figure 10-4: A mesh eliminates bottlenecks and single points of failure

A mesh isn't perfect. The biggest problem is that it doesn't scale very efficiently. As you can tell, as more switches are added, the number of ports required to connect to all the available switches will use up most of the ports on each of the switches. Given that limitation, the mesh's strength is a good choice for midsize SANs with five or fewer switches requiring a maximum guaranteed uptime and optimal performance.

Scalability and redundancy are brought together in the next model, as shown in Figure 10-5. To ensure a redundant data path, each switch is connected to two other switches. Each switch has two different paths through the SAN, thereby eliminating a single point of failure. This configuration is an ideal solution for enterprise-class SANs.

Figure 10-5: Connecting to two other devices reduces bottlenecks and increases scalability

Management

For a SAN to work at its peak, it must use centrally managed devices. SAN management is mainly a security device that ensures servers see only the intended devices and storage arrays, reducing the chance for data corruption. There are two basic ways to manage your SAN's hardware: port-level zoning and logical unit (LUN)-level zoning.

• Port-level zoning is similar to a virtual LAN (VLAN). Port zoning partitions devices based on which ports they are using on the hub or switch. Attached nodes won't be able to communicate unless individual ports are shared in a common zone.

• LUN-level zoning is similar to port-level zoning; however, it increases the granularity, thereby making it possible to partition nodes by their device ID. LUN-level zoning gives you more flexibility when it comes to communicating with devices on the edge of a SAN.

Note 

We'll talk more about zoning with Cisco devices later in this chapter and include a discussion of virtual SANs (VSANs).

Data storage can be an ever-changing environment. Storage needs may differ drastically from one day to the next. Happily, however, managing a SAN is helpful, because it gives you the ability to dynamically allocate storage to the different pools without having to reboot the servers in your storage cluster. In addition, you can add more storage to your SAN and reallocate it as you wish, again with no interruption.

Storage Options

When it comes to deciding what you'll use as the storage component of your SAN, there isn't an abundance of options. An easy option, especially if you already have SCSI RAID or disk shelves, is to buy an external SCSI-to-fiber bridge. A bridge will allow you to connect almost any SCSI device to your SAN. The downside of this is that you waste all the speed that a natively attached fiber device would deliver. Bridges push between 15 Mbps and 40 Mbps.

This leads to the second option: native fiber-attached storage. Fibre Channel storage is becoming more and more popular and-as is the case with technology-the price is coming down. Fiber storage can push 4 Gbps, but even though prices are coming down, they aren't nearly as inexpensive as a SCSI RAID solution. Costwise, the hard drives are on a par with SCSI drives. It's the external Fibre Channel RAID controllers, at anywhere from $8,000 to $50,000, that jack up the price.

Backup

Fiber-optic backups are just the thing for administrators who manage networks with copious amounts of data. Not only can they offload backups from the network, but they can also share a single library among multiple servers scattered throughout several different departments. Furthermore, resources can be allocated to the departments that have the greatest backup needs. In addition to drives, tape libraries can use SCSI-to-fiber bridges. Though tape libraries don't come close to the speed of drives and may seem a waste of fiber resources, many implement tape libraries into their SANs because backups are easier to perform.

Routers are just starting to come into their own in the world of SANs. Routers in a SAN are intelligent devices that can execute a direct disk-to-tape backup without the middleman of the server processing the information first. As you can imagine, backing up information without taxing the server not only releases the server to perform other tasks, but it also reduces backup times by removing any bottlenecks that might occur as the data filters through the server. Furthermore, routers have the technology integrated with them to handle error recovery, as well as the capability to report problems to the backup software.