4.3 Fabrics

A Fibre Channel fabric is one or more fabric switches in a single, sometimes extended, configuration. Fabrics provide full bandwidth per port approximately 200MBps full duplex for 1Gbps links, and 400MBps full duplex for 2Gbps links. Although the introduction of new devices to an arbitrated loop further divides the shared bandwidth, adding new devices on each port of a fabric increases the aggregate bandwidth. An eight-port fabric with four initiators and four targets, for example, can support four concurrent 200MBps conversations at 2Gbps, or a total of 800MBps throughput (1600MBps if full duplex applications are available). As shown in Figure 4-10, a fabric switch is optimized for direct connections between switch F_Ports and device N_Ports.

Figure 4-10. A fabric switch provides full bandwidth per port

The switching mechanism typically used in fabrics is called cut-through. Cut-through technology is also used in Ethernet switches to speed packet routing from port to port. When a frame enters a fabric port from an attached N_Port, cut-through logic examines only the link-level destination ID (D_ID) of the frame. In Fibre Channel, the destination ID is the 24-bit port address of the target N_Port. A routing decision is based on the D_ID, and the frame is switched by internal routing logic to the appropriate port. Cut-through increases performance by reducing the time required to make a routing decision. Because the D_ID resides in the first 4-byte word of the frame header, a routing decision can be made immediately as the frame enters the fabric port. Alternatively, the store-and-forward algorithm employed by some switch technologies requires that the entire frame be buffered before a routing decision occurs.

The number of frame buffers offered on each F_Port is an important consideration for switch selection. During periods of congestion, the fabric may not be able to simply transmit a frame on the destination port, especially if two N_Ports are sending data to the same destination. Depending on the class of service, the fabric may be forced to abandon frames it cannot process. The ability to queue multiple frames reduces this possibility and enhances performance for both transmitting and receiving N_Ports.

Fabrics, unlike arbitrated loop, are not restricted to an address space based on transmission word running disparity (AL_PAs). Instead, fabric-attached N_Ports are assigned a 24-bit port address, which theoretically allows for 2²⁴, or more than 16 million, possible addresses. The address allocation scheme introduced by the Fibre Channel switch fabric (FC-SW) standard reduces this number somewhat to around 15.5 million. Because fabric products available in the market provide 8 to 256 ports, this reduction from 16 to 15.5 million addresses has not yet caused an uproar among end users.

As shown in Figure 4-11, the FC-SW standard divides the 3-byte port address into three 1-byte fields. The most significant byte is the domain, the next significant is the area, and the least significant is the port. The 8-bit domain field has a number of reserved addresses, leaving 239 available for assignment to individual switches. This allows each switch in an extended fabric to have a unique identifier and also limits the largest possible fabric to 239 interconnected switches. In practice, most SANs do not connect more than a few switches in a single fabric. The 8-bit area field provides 256 area addresses. You can use these addresses to identify individual FL_Ports that support loops, or groups of F_Ports. An example of the latter is a multiport card installed in a switch chassis, which can be governed by a separate part of the routing engine. Finally, the 8-bit port field provides 256 addresses for identifying attached N_Ports or NL_Ports. The division of the 24-bit address space into domain, area, and port provides enhanced switching performance by enabling a routing decision to be made on the basis of a single byte. If an incoming frame, for example, has a domain address intended for a different switch in the fabric, the routing engine can forward the frame to the appropriate interconnect or E_Port without processing the entire 24-bit address.

Figure 4-11. Fabric addressing

Fibre Channel switch products offer ports ranging from 8 ports for departmental fabrics to 128 or more ports for larger enterprise-level directors. Because fabrics are extended via switch-to-switch links on E_Ports, a potential bottleneck could occur during periods of high activity between switches. You could resolve this problem by providing multiple E_Ports between two interconnected switches and a load-balancing algorithm to distribute traffic. Such a solution, however, might result in out-of-order delivery of frames from source to destination because a sequence of frames could take different paths through the fabric. Consequently, some implementations allow only a single active link between two switches, whereas others provide additional logic to prevent out-of-order delivery. Trunked 2Gbps E_Port connections can be used in some products to build higher-speed interswitch connections, and 10Gbps Fibre Channel will provide even higher performance interswitch links.

You can configure a complex fabric of multiple interconnected switches in a mesh topology, with alternative data paths from one N_Port to another through the fabric cloud, as depicted in Figure 4-12. Redundant paths through the fabric are desirable, because the failure of one path will allow traffic to be routed through alternative paths. As in the example just cited, concurrently active alternative paths could result in out-of-order delivery of frames. A meshed fabric can avoid this problem by employing spanning tree, an IEEE 802.1 standard commonly used in LAN bridges and routers. The spanning tree protocol is used by network devices to determine optimum routes through a network and to hold redundant links in reserve. Alternative paths are activated only when a primary path fails. Spanning tree thus avoids duplication of active paths through the network and, in the case of a meshed fabric topology, allows in-order delivery of frames.

Figure 4-12. A partially meshed fabric of four Fibre Channel switches

Because fabrics must support N_Ports, public arbitrated loop, and switch-to-switch interconnect, they are considerably more complex than loop hubs. This is reflected in the per-port cost of fabrics compared with that of hubs, although competition and the introduction of efficient ASIC (application-specific integrated circuit) technology are driving the per-port cost down. Considering that fabrics and loop hubs sometimes tie together hundreds of thousands of dollars' worth of servers and storage and that these systems support the business of entire enterprises, Fibre Channel interconnect products are the least expensive component of any storage network.

Fabrics provide a number of services that rationalize activity on the transport and facilitate adds, moves, and changes of attached devices. Fabric login services allow N_Ports and public loop NL_Ports to register with the fabric for communication. Simple Name Server (SNS) functions streamline the process of discovery between communicating ports. And State Change Notification (SCN) allows all fabric devices to keep current with topology changes caused by the introduction or removal of devices.

4.3.1 Fabric Login

When a fabric-capable Fibre Channel node is attached to a fabric switch, it performs fabric login. Like port login, FLOGI is an extended link service command that establishes a session between two participants. In the case of FLOGI, an association or login session is created between the N_Port or public loop NL_Port and the switch. An N_Port sends a FLOGI frame containing its node name, N_Port name (World-Wide Name), and service parameters to a well-known address of xFFFFFE. A public loop NL_Port first opens the destination AL_PA x00 before issuing its FLOGI request. In both cases, the login handler in the switch (at address xFFFFFE) accepts login by returning an accept (ACC) frame to the sender. If the fabric does not support certain service parameters requested by the N_Port or NL_Port, it sets bits in the ACC frame to indicate as much.

An N_Port uses a 24-bit port address of x000000 when it logs in to the fabric. This allows the fabric to assign the appropriate port address to the device, based on the domain/area/port address format. The newly assigned address is contained in the destination (D_ID) field of the ACC response.

A similar process is used for a public loop NL_Port, except that the least significant byte is used to assign an AL_PA, and the upper two bytes constitute a fabric loop identifier. Because the NL_Port has just emerged from a loop initialization (LIP) process before performing fabric login, it will already have a previously derived AL_PA. It is up to the fabric to determine whether it will accept this AL_PA. If it does not, a new AL_PA is assigned to the NL_Port, and that causes the fabric to launch another LIP process on its local loop segment. This ensures that the fabric-assigned AL_PA does not conflict with any previously selected AL_PAs on the loop. During the subsequent loop initialization sequence, the NL_Port will now have a fabric-assigned address that it can select during the LIFA phase.

Fabric login establishes the initial relationship between the switch and an attached device. Reliance on the fabric to automatically assign a network address and to negotiate capabilities between the switch port and the device is meant to streamline the installation and management of the fabric. As discussed later, however, administrators still need to manage the fabric to avoid configurations that may inadvertently result in duplicate addressing.

4.3.2 Simple Name Server

As you saw in the discussion of arbitrated loop, initiators on the loop begin the process of target discovery by performing port logins to all 126 AL_PA device addresses. At gigabit speeds, this process occurs fairly quickly and allows each server to find target disks whenever a topology change occurs.

This process is untenable in fabric environments, however, because the potential address space is more than 15.5 million addresses. Even at gigabit speeds, this would take some time. Fabrics rationalize the process of device discovery by providing a name server function in each switch. The SNS is a database of objects that allows each fabric device to register or query useful information, including the name, address, and class of service capability of other participants. To keep the database compact, some objects such as the type of FC-4 protocols supported by a registrant are coded into a positional bit map.

The name server is accessed by performing a port login (PLOGI) to a well-known address of xFFFFFC. An N_Port or public NL_Port registers with the name server after performing PLOGI to it and then terminates the session. The device may register values for some or all of the database objects, but the most useful are its 24-bit port address, 8-byte port name, 8-byte node name, class of service parameters, FC-4 protocols supported, and port type (N, NL, etc.). Devices that support IP can also register their IPv4 or IPv6 IP addresses. Because the name server contains a table of both 24-bit addresses and the corresponding 64-bit WWNs, one device can find another based on either category.

The fabric name server makes it possible for a file server, for example, to begin discovering disk targets by inquiring for a list of all port addresses registered with the switch. This would relieve the file server of the need to poll 15.5 million addresses. The file server could then perform PLOGIs with each reported address to discover, via upper-layer commands, which devices were SCSI-3 targets. Alternatively, because the name server also contains information on FC-4 protocol support, the file server could PLOGI only those port addresses that reported SCSI-3 support.

The name server in each Fibre Channel switch enables initiators to discover targets. When two or more fabric switches are connected via E_Port, an initiator on one switch might also need to access storage resources connected to the other switches. To discover those fabric-wide resources, the switches themselves must exchange SNS information. This exchange is performed using the Class F protocol over the expansion port connections and assumes that each switch's SNS is large enough to accommodate additional entries. In very large fabrics, this requires additional memory for SNS data and may prolong the convergence time of the fabric as each switch updates its name server.

4.3.3 State Change Notification

Because fabrics are dynamic environments and may undergo topology changes when new devices are attached or previously active devices are taken off line, it is useful if notification of changes can be provided to other participants. This function is provided by State Change Notification (SCN) and a variant, Registered State Change Notification (RSCN). SCN and RSCN are voluntary functions in that an N_Port or NL_Port must register its interest in being notified of network changes if another device alters state.

The original SCN service allowed an N_Port to send a change notification directly to another N_Port. RSCN does not allow this because a direct N_Port-to-N_Port notification would not be seen by the remainder of the registrants. As a fabric service, RSCN allows all devices that have registered for notification to be informed if a topology change occurs. Because storage targets are passive recipients of commands from initiators, there is no need for targets to be alerted when devices enter or leave the fabric. Servers, by contrast, want to be notified when resources are no longer available or when new resources have been added. Change notification thus provides for fabrics a function that loop initialization intrinsically provides for arbitrated loop.

4.3.4 Private Loop Support

Fabrics support public arbitrated loop devices via FL_Ports. The FL_Port has the highest-priority AL_PA, x00, always wins arbitration, is always temporary loop master during loop initialization, and provides the interface between the loop segment and the rest of the fabric. To communicate in a fabric environment, however, loop devices must be fabric-capable that is, they must perform fabric login following loop initialization.

Fibre Channel standards do not support private, nonfabric loop devices on a fabric topology. However, this has not prevented fabric vendors from devising means to accommodate the large population of private loop devices in the market. Allowing private loop-only devices to participate in a switched topology has obvious benefits for storage networks that have been steadily deploying arbitrated loops over the past several years. Although upgrading a file server with a fabric-capable HBA is affordable, performing a forklift upgrade on a private loop disk array is not. Finding a means to incorporate these resources into a fabric is therefore a very attractive option.

Private loop support on a switch implies the need for additional intelligence to monitor the activity of the private loop devices and to ensure that the existence of the fabric is transparent to them. Because the private loop devices are not aware of the fabric's presence, private loop support on a switch is sometimes described as stealth, or phantom, mode. Various implementations are available.

A segmentation strategy allows private loop devices to be dispersed on specific fabric ports, as shown in Figure 4-13. Although vendors may restrict the total number of private loop devices on a single port, the allocation of devices per port is usually determined by bandwidth and traffic considerations. Servers, for example, could be assigned to shared ports, with large JBODs (representing multiple AL_PAs) assigned to other dedicated ports. What was previously a single, continuously shared transport would thus be segmented into a series of smaller arbitrated loop segments, some of which would have only one private loop device per port. Segmentation transforms what was previously a physical arbitrated loop into a virtual arbitrated loop. Because the private loop devices are not aware of the fabric, they arbitrate, open, transfer, and close transactions as before. A server on its own port, for example, arbitrates for access as usual, but because it is the only node on that loop segment, it has no contenders and immediately wins. When it issues frames to a destination AL_PA, the switch logic intervenes and routes the frame directly to the switch port where that AL_PA resides. By this means, segmentation provides the high-speed bandwidth of a fabric without requiring fabric support. Segmentation also allows multiple conversations to occur simultaneously on the same virtual loop.

Figure 4-13. Private loop support on a fabric switch

A translational strategy allows private loop devices to communicate with fabric-capable devices. You can implement this strategy by manipulating the upper two bytes or loop identifier of the private loop device's port address. On the private loop segment, the loop identifier is x0000, indicating that the device is nonfabric. At the FL_Port, the fabric can proxy a loop identifier on behalf of the private loop device and create an entry in the SNS, thus allowing fabric-capable devices to address it. This manipulation implies a passive role for the private loop device, because it would require considerable overhead to allow private loop servers to discover and communicate with fabric-capable storage. Translational private loop support is thus better suited to the access of private loop storage by fabric-capable servers.

4.3.5 Fabric Zoning

Fabric zoning allows you to segregate devices based on function, departmental separation, or potential conflicts between operating systems. Windows NT's habit of claiming all available storage resources, for example, makes it highly desirable to isolate UNIX storage from NT storage. Without some means of separating resources, putting marketing department information on the same fabric with engineering might inadvertently give engineers a peek at the overly aggressive plans of marketing managers. Zoning resolves these potential problems by assigning devices or ports to separate zones or groups, and enforcing separation via selective switching.

Zoning can be implemented on a port-by-port basis, thereby providing greater security, or by 24-bit port address or 64-bit WWN. The latter variation usually requires a dedicated server to configure and maintain discrete assignment of subgroups. Port-based zoning is less involved, because you can use filtering logic in the routing engine to maintain the assignment of ports to one or more zones.

A classic application for zoning is tape backup for heterogeneous operating systems, as illustrated in Figure 4-14. Windows NT servers and storage can be assigned to, say, zone A, whereas Sun Solaris servers and storage are assigned to zone B. This segregation prevents NT from accessing the Sun storage arrays. Both NT and Solaris servers, however, could be assigned to a common zone C, which would include the tape subsystem. Zone C would allow both servers to back up or restore from tape.

Figure 4-14. Port zoning on a fabric switch

Port zoning is relatively more secure than WWN-based zoning because an intruder cannot spoof port zoning by manipulating Fibre Channel frame information. The zone, however, stays with the port and not the device. So if someone inadvertently moves a server from one port to another, the server is no longer part of its former zone. In WWN-based zoning, the zoning stays with the device. This facilitates movement of devices in the fabric but is vulnerable to being spoofed.

Zoning implementations in fabric switches may be proprietary, although switch interoperability has required common tagging methods so that a zone created on one vendor's switch can be recognized by another vendor's switch. Zone merging is now part of the switch-to-switch protocol and, like SNS exchange, requires additional handshaking between switches before the fabric can become stable and operational.