5.9 Fibre Channel Extension Products

Because organizations often have multiple regional or international sites, the ability to access storage data over distance has gained increasing importance. In addition to disaster recovery, wide area storage applications such as consolidated tape backup, resource sharing, and content distribution require a means to extend Fibre Channel-originated traffic beyond the central data center.

Several solutions are currently available for Fibre Channel extension, although the high ground for block storage data over distance has been claimed by IP-based storage products. For native Fibre Channel extension transport methods that require no protocol conversion the most common extension products rely on dark fiber and dense wave division multiplexing or on encapsulation of Fibre Channel frames using IP tunneling.

5.9.1 Fibre Channel Extension Using DWDM

Dense wave division multiplexing (DWDM) is a physical transport that leverages optical signaling to send numerous data streams concurrently over a single optical link. Light traversing an optical cable is composed of various modes or wavelengths of light. If a data stream is associated with a specific wavelength of light at the point of transmission, it can be separated from other data streams at the receiving end, as illustrated in Figure 5-16. With current technology, as many as 64 concurrent streams can be supported, each riding its own wavelength.

Figure 5-16. DWDM transports each data source on a separate wavelength

DWDM requires dedicated fiber cabling, commonly referred to as dark fiber. Dark fiber is any unused optical pair of an installed cable run. In contrast, lit fiber is already carrying Ethernet, ATM, or some other transport. Carriers with unused fiber are always willing to sell access for a price, but dark fiber is not always available in all areas. Alternatively, an organization could run its own fiber at some expense, providing it has the right of way from source to destination.

DWDM can carry any protocol but has become linked to Fibre Channel extension because of its ability to drive longer distances more efficiently than traditional longwave optics and single-mode cable. Still, even with DWDM, performance of Fibre Channel switches over distance falls dramatically beyond metropolitan distances.

Fibre Channel fabric switches were originally designed for data center applications within a fairly narrow circumference (500m with multimode cabling and shortwave optics). With adjoining equipment in close proximity, port buffer requirements are minimal. A fabric switch port, for example, can buffer 3 or 4 frames before they are placed on the cable for transmission. As the switch receives additional credits from the receiving end, it can empty its buffers onto the wire. Under congestion, a switch may have to buffer more frames, typically 32 frames, before frame discard occurs. To empty its buffers, it would have to receive as many as 32 credits from the destination device.

In long haul applications, speed-of-light latency over distance increases the wait time for credits or acknowledgments to return from the receiving end. Depending on the distance, a typical fabric switch port could send its maximum allowed frames and then sit idle for milliseconds as it waited for credits to return. Even with full gigabit bandwidth available on the link, the credit limitation hinders performance, and throughput continues to fall as the distance is extended.

Some fabric switch vendors have attempted to overcome credit starvation by pooling buffers from adjoining ports to serve the long haul port. Although this technique enables the switch port to provide better performance, the contributor ports are unusable for normal node attachment. The customer is thus asked to pay for the inadequacies of the original switch design. Given the low cost of memory, the better solution is to simply design in more buffers per port.

Fibre Channel extension with DWDM can be used for either switch-to-switch or switch-to-node connectivity. From the standpoint of the fabric switch, the intervening DWDM equipment and long haul cable plant are transparent. Consequently, E_Ports or F_Ports can be connected to the DWDM infrastructure as if it were a straight run fiber cable. For E_Port connections, switch-to-switch behavior including principal switch selection, route information and SNS exchange, zone merging, and so on now occurs across an extended link. As long as the link is stable, this stretched E_Port connection presents no major difficulties. If the link fails, however, the effect is the same as pulling the cable between two adjacent fabric switches. Each would undergo fabric reconfiguration, with all storage conversations suspended until each fabric stabilized.

5.9.2 Fibre Channel Extension Using IP Tunneling

Because of the higher cost of dark fiber compared with other communications services, use of the more common and more affordable IP network services is an attractive option for Fibre Channel extension. Fibre Channel over IP (FCIP) is an IP tunneling solution favored by Fibre Channel vendors because it perpetuates acquisition of Fibre Channel switches on both sides of an extended link. Alternatively, two native IP storage protocols (iFCP and iSCSI) can also be used to extend Fibre Channel-originated traffic. Because iFCP and iSCSI are more properly IP storage technologies, they are discussed in Chapter 6.

FCIP tunneling performs a simple task, although the cost of FCIP products often does not reflect it. As shown in Figure 5-17, an FCIP device typically attaches by E_Port connection to the source Fibre Channel switch. Fibre Channel frames destined for the remote end are wrapped in IP datagrams and sent across the IP network. At the receiving end, the IP datagrams are stripped off, and the original Fibre Channel frames are delivered to the E_Port of the receiving fabric switch for routing.

Figure 5-17. Using FCIP tunneling to span distance between fabric switches

Because FCIP implements a standard E_Port connection over distance, normal Fibre Channel switch-to-switch protocols are passed over the wide area link. This creates a single Fibre Channel fabric, which is now stretched over tens or hundreds of miles. Disruptions at one site may propagate to another, or a break in the wide area link may trigger a fabric reconvergence.

With the caveat of stretched E_Port connections, FCIP tunneling is suitable for applications that require connectivity only between two sites. In current product offerings, some FCIP devices may provide additional buffering to compensate for buffering limitations of fabric switches and may also provide data compression to maximize use of the available WAN bandwidth. In addition, some FCIP products incorporate WAN interfaces for connecting directly to carrier services, whereas others rely on IP routers to connect to the cloud.

FCIP is a point-to-point wide area connection. A pair of FCIP devices is therefore required for each remote link, and each remote site assumes the existence of a fabric switch in addition to end nodes.

5.9.3 Fibre Channel WAN Bridging

Still being defined in standards, Fibre Channel WAN bridging offers another solution for SAN extension. A Fibre Channel WAN bridge transports Fibre Channel traffic over non-Fibre Channel topologies such as ATM. Connection to a local fabric is via a B_Port (bridge port) interface on the bridge. B_Ports, which provide a subset of E_Port protocols, either transparently pass through switch-to-switch Class F frames or engage in the fabric building process as an additional switch.

WAN bridging between two Fibre Channel SANs may result in a single fabric spread over distance. In this case, the WAN bridges pass E_Port traffic between fabric switches, and a common address space with a unique Domain_ID is established for the dispersed SAN. Alternatively, the WAN bridge can engage in E_Port behavior with its local fabric switch, creating an autonomous Fibre Channel region, as shown in Figure 5-18. The notion of autonomous regions is meant to overcome the inherent vulnerability of extended SANs to disruptions. If instead of being joined by an E_Port into one large fabric, each site is an autonomous region, then propagation of disruptions can be limited. Establishment of autonomous regions still requires that each fabric switch have a unique Domain_ID. Because this is usually accomplished automatically through principal switch selection, implementing autonomous regions will require manual administration of addresses for each switch.

Figure 5-18. Autonomous regions created with Fibre Channel WAN bridging