Fibre Channel SANs

The HBA you use to connect a device to a storage network determines the physical connection type or medium as well as the transport protocol you use. In Chapter 11 we considered some of the most common HBAsin particular, various forms of SCSI and Fibre Channeland briefly described their utility in building SANs. Both SCSI and Fibre Channel are used to connect servers to DAS systems. The small number of possible nodes on a SCSI bus and the short-run lengths of that medium limit SCSI to only DAS storage. But Fibre Channel is another story.

See "HBAs and RAID Controllers," p. 593.

Fibre Channel is, for the moment at least, the dominant interconnect technology used to create SANs. Although Fibre Channel SANs were originally in the form of arbitrated loops, for the past several years Fibre Channel fabric SANs have dominated. A Fibre Channel bus can support 126 drives or nodes per bus, with a theoretical 40 to 60 nodes in practice, which limits a FC-AL to small storage networks and more often to the internal connections in large storage servers. When you attach a Fibre Channel HBA to a fabric switch, the fan out to other fabric switches allows for 2²⁴, or 16 million, nodes. Although Fibre Channel is still expensive and somewhat difficult to connect correctly, it still accounts for most of the sales in the SAN HBA market.

More About Fibre Channel

Fibre Channel is described as having two layers: a lower signaling level and an upper level of services and protocol mapping. The signaling layer includes the physical interface and media layer (FC-0) on which a transmission or wire protocol is layered (FC-1). Signals are frames that travel in a framing and signaling protocol layer that form the top of the signaling layer (FC-2). In the upper layer are common services (FC-3) on which there is a protocol mapping layer (FC-4), which is where the framing rules are applied. The FC-3 layer transports information about fabric network device discovery, RAID levels, the type of encryption and compression being used. The FC-4 layer allows for mapping bus protocols like SCSI, IP, and VI (virtual interface).

More important in determining compatible hardware is the Fibre Channel class levels. A Fibre Channel class determines the flow control in the FC-2 layer, and that is how Fibre Channel components negotiate with one another. You need to match device classes if you can to optimize performance. Currently, six classes are defined:

• Class 1 This class is used for end-to-end connections. Because each frame is verified, in Class 1 there is no negotiation; each device in the point-to-point connection has the full attention of the wire. Class 1 isn't used on any shared storage system because it is a closed system.

• Class 2 This class is a frame-switched connection that allows for a shared connection in a fabric. Although Class 2 verifies all frame delivery, it does not require that frames be ordered in any way. Because frames aren't sequential, Class 2 Fibre Channel can't be used for SCSI data, which must be sequential. Some vendors' Class 2 switches support sequential delivery as a proprietary feature. You can tell which switches those are because they support SCSI over Fibre Channel.

• Class 3 This class is another frame-switched connection, but here the overhead of frame acknowledgement is removed. It is up to the target and source systems to determine whether a transmission has been successfully accomplished. Class 3 uses the buffer-to-buffer flow control method that is described later in this section. Class 3 Fibre Channel also doesn't sequence frames, but it does include a broadcast feature that can send simultaneous traffic to more than one device.

• Class 4 This class allows for fractional bandwidth allocation on a fabric, using what is called a virtual circuit. Thus, Class 4 allows for a shared connection but offers some of the advantages of Class 1 Fibre Channel. Class 4 is just coming into use.

• Class 5 This class is a definition of isochronous (same time) and just-in-time service. It is unclear whether Class 5 will ever be implemented in products.

• Class 6 This class is a proposed multicast fabric service with dedicated connections.

Note

You can find a detailed tutorial on Fibre Channel standards at www.recoverdata.com/fc_tutorial.htm#Class%204.

The most common type of Fibre Channel in use today is the 100MBps variety. Earlier standards of 25MBps and 50MBps were used, and today they are referred to as quarter- and half-speed Fibre Channel. These slower speeds exist as part of legacy deployments. The road map for Fibre Channel includes 200MBps HBAs and switches (which are just starting to be employed), with 400MBps and 1GBps speeds promised. Given that competitive technologies are being introduced in the 1GBps range and above, it is likely that vendors will ignore 400MBps and jump directly to the 1GBps (and faster) speeds.

In Fibre Channel connections, cables and connectors are passive devices. The signal is transmitted and received by an electronic component called a transceiver. Each Fibre Channel connection contains two transceivers, and traffic down each wire flows in one direction. Thus one wire transmits data from one port to another port, while on the second wire connecting these two ports, data is transmitted in the opposite direction. Therefore, Fibre Channel avoids a number of issues that plague other networking technologies, such as Ethernet: signal contention and interference. A two-wire transmission scheme also simplifies the connectors so that they are easier to design and implement. The real advantage that Fibre Channel offers is that a loop or fabric network eliminates a single point of failure because most Fibre Channel HBAs used are dual homed, to take advantage of loop topology.

Switches

Although you can buy Fibre Channel hubs and routers, most Fibre Channel interconnect devices sold are switches. On SANs, switches can be small and unmanaged, but most often they are managed devices with some intelligence. A Fibre Channel SAN switch has the following:

• Onboard processing

• The capability to route storage traffic

• The capability to be both discoverable and manageable through SNMP

• The capability to be programmable through the vendor's API

Entry-level Fibre Channel switches go up to around 16 ports. Brocade (www.brocade.com) dominates the market for entry-level switches with its SilkWorm line. Brocade also does well in the larger workgroup switches space; these devices are most often defined as having 16 to 64 ports. Figure 12.8 shows the highly regarded 16-port SilkWorm 3850 fabric switch, which is a 2Gbps-per-port autosensing switch. This switch is compatible with the slower 1Gbps devices. Brocade's switches go up to the 4Gbps speed and to the higher end of the switch market64 ports and beyondwhere switches are referred to as director-class or director-level switches.

Figure 12.8. The Brocade SilkWorm 3850 16-port 2Gbps Fibre Channel switch.

Brocade is so well entrenched in this marketplace that its API has become something of a standard. Nearly all software management programs, autodiscovery software, and other SAN software offerings support Brocade's products.

Brocade does have competitors, and it is in the workgroup and director-class markets that there is vigorous competition. The company in the director class with the most market share is McData, with its Intrepid switches. Other significant players are Cisco, with its MDS line of switches, and QLogic. Cisco is the 800-pound gorilla of the networking market, but although the company was expected to become a major player in storage networking, it has moved slowly in this marketplace.

A simple FC-AL loop topology can be created by connected all devices to a nonswitching Fibre Channel router or hub. Connecting two FC-ALs together requires having a switched path, and the simplest way to create that is to connect a switching hub on one FC-AL through a Fibre Channel bridge to the second FC-AL. When you connect multiple loops, they share the address space but not the bandwidth: Each loop still has its full rated speed available to its connected devices.

A fabric topology requires (as you saw earlier in this chapter) at a minimum a switch into which all devices are connected. It's easier to build a fabric than a loop topology in that you can connect two switches to one another to grow your network.

Port Types

To understand how SAN connections are made, you need to know a little bit about switch port definition. Each connection on a Fibre Channel switch is called a port, and each connection on the Fibre Channel HBA is also called a port. Any Fibre Channel device that you hook up using a Fibre Channel cable to a Fibre Channelbased SANbe it a hub, switch, bridge, or systemis a port on the storage network. The cable attaches to one port on one end and one port on the other end so that for each port there is a single cable connection. The entire system attached to a Fibre Channel HBA is referred to as a node.

Fibre Channel defines eight port types:

• B-ports are bridging ports that can connect to additional networks.

• N-ports are used in fabric switches where they both initiate and receive frames.

• L-ports are used in loops.

• NL-ports can be used for both fabrics and loops, and they can be used to connect through a loop to a fabric.

• E-ports are switch ports that connect to other switch ports.

• F-ports are switch ports that connect to N-ports.

• FL-ports are switch ports that connect to NL-ports.

• G-ports are common, or "generic," ports that can be used to connect to E, F, or FL-ports.

All Fibre Channel ports and connections are passive; all the addressing and packaging of frames is done in the controller. Fibre Channel provides a heartbeat so that regular signals are sent between N-ports and F-ports and are received to indicate the port status. Because L-ports were designed to work in loops, they both initiate and control traffic.

Addressing

Fibre Channel has its own addressing and naming scheme, just like Ethernet does. To create a unique network name, Fibre Channel uses the following:

• World Wide Name (WWN) The WWN is a 64-bit identification number that is assigned for each Fibre Channel device by the manufacturer and burned into the device's ROM. You can think of the WWN as Fibre Channel's equivalent of an Ethernet MAC address; when you discover a Fibre Channel port in software, you can inventory what's attached to it using the WWN.

• Port address A port address is a unique 24-bit address that is assigned to a port on a network. You can think of a port address as the Fibre Channel equivalent of an IP address in a TCP/IP network. The 24-bit address space defines 2²⁴, or 16 million (16,777,216), possible addresses. Assignment of port address numbers is determined by the person or organization setting up a Fibre Channel SAN.

• Arbitrated Loop Physical Address (AL-PA) AL-PA is used on loop topologies to define an addressable and unique address. Because loops have a much lower number of connected nodes, an 8-bit address scheme allowing 256 possible addresses, which is the theoretical limit.

Traffic Flow

To mitigate traffic over the network, Fibre Channel manages traffic by using a scheme called buffer credits. Each port is given a certain budget of traffic, and when that budget is used up, traffic is switched to the next port. Buffer credits are issued for end-to-end flow control where the destination N-port, L-port, or NL-port sends an acknowledgement that the frame arrived correctly and an additional credit is given to that port because it is now waiting for data. A second form of flow control, called buffer-to-buffer, manages a set of credits between adjacent ports. In this form of traffic control, the target port issues a receive ready signal to the sending port.

On a pure FC-AL network, all ports are L type, and this type of loop is called a private loop. No other network can access a private loop because it is a closed system. Many FC-ALs are built so that some devices can remain private while other devices are viewable to a connecting network. This sort of loop is called a public loop, as shown in Figure 12.9. On a public loop, private devices are still connected to L-ports, public devices are connected through NL-ports, and the arbitrated loop connects to another network via an FL-port connection to a switch.

Figure 12.9. A public loop.

For a Fibre Channel loop, the arbitration scheme used is similar to the one that SCSI uses to perform bus arbitration. In this scheme, the port with the highest priority gets priority to send and receive frames. Unlike SCSI, which determines priority based on electronics, on an FC-AL, the priority is done in the command language. As an L- or NL-port on an FC-AL requires loop access, it starts to issue commands that indicate it is ready. As the commands circulate the loop, each node determines its priority in relationship to the node requesting loop control; either that node takes command or passes the control command on.

At some point, the loop becomes available to the arbitrating port, traffic flows or an exchange occurs, and then another higher-priority node assumes command of the loop. Exchanges are a set of data sequences, where the data is sent as a frame and in a prescribed order (thus a sequence). Fibre Channel SANs use a rather small frame size, typically 2KB.

A Fibre Channel frame is similar in construction to other packet frame type networking protocolsthat is, the frame has start and end markers, a header that defines the frame, addresses, data, error connection, and a validation data set that performs both acknowledgement and data recovery. A frame is constructed like an envelope. It encapsulates data, just like in other wire protocols, so that Fibre Channel can be constructed so that it maps to other upper-level protocols, such as SCSI, IP, HIPI, FICON, ESCON, 802.2, and Virtual Interface Architecture (VIA).

With a Fibre Channel fabric, a different kind of process is used to determine flow control. A set of nodes and process logons communicate the status of different port types in a fabric network. As each node logs on to the network, initiator and target ports perform a node logon, which establishes the data transfer connection. When different node types exist, the option process login can be used to establish the type of protocol that the transferred data uses. Fabric networks use the FCP (Fibre Channel Protocol) serial SCSI protocol. This kind of process happens for any type of Fibre Channel connections, even direct attached Fibre Channel.

Fibre Channel is almost always installed as a switched fabric for networked storage. However, that doesn't mean that Fibre Channel loops are going to disappear anytime soon. Internal Fibre Channel connections within storage devices such as large storage arrays will continue to be used for the foreseeable future.