Section 4.4. Fibre Channel Topologies

   

---

4.4 Fibre Channel Topologies

Sections 4.4.1 through 4.4.3 describe the different topologies that enable interconnection of all the various elements that constitute a Fibre Channel SAN. From least to most complicated, the three different topologies are point to point, arbitrated loop, and switched fabric.

4.4.1 Point to Point

Fibre Channel may be configured with a point-to-point topology. In this case typically a server is connected to a dedicated storage subsystem. There is no data sharing. Figure 4.1 illustrates a point-to-point network.

Figure 4.1. Fibre Channel Point-to-Point Topology

To implement a Fibre Channel point-to-point topology, the minimal equipment needed consists of a server, a Fibre Channel adapter (called a host bus adapter ) and a storage device (e.g., disk or tape) with a Fibre Channel interface.

4.4.2 Arbitrated Loop

A loop is simply a logical connection of devices such that data flows in a logical ring fashion around the loop. An arbitrated loop is one in which a protocol specifies how a node may seek permission to accomplish data transmission. A Fibre Channel arbitrated loop (FC-AL) can be implemented with various storage devices (such as disk and tape), servers, HBAs, and an interconnect device. This interconnect device can be a Fibre Channel hub or switch. Both of these are described in Section 4.7.4. For now, suffice it to say that the interconnect device plays an important role in the cabling, as well as in the operations and management, of the loop.

Figure 4.2 shows an FC arbitrated loop. The configuration is analogous to a physical star and logical ring, just as in the token ring LANs. And also just as in token ring LANs, data flows in only one direction around the loop. Unlike token rings, however, a device requests permission to communicate from the hub, rather than wait for an empty token to come around.

Figure 4.2. Fibre Channel Arbitrated-Loop Topology

Fibre Channel commands enable negotiation and access to the loop for transmission purposes. Commands are also available to assign arbitrated-loop port addresses (AL-PAs) to the various nodes in the loop. Each node on the FC-AL loop has bypass circuitry to take itself out of the loop and maintain loop continuity in cases of errors.

FC-AL loops can address up to 127 ports (of the NL variety; see Table 4.2 later in this chapter) by the way AL-PA addresses are specified. One of these ports is reserved for connection to a fabric switch (described in Section 4.7.4.3), leaving 126 possible addresses that can be assigned to nodes. For practical performance reasons, however, typical loops contain 12 or fewer nodes, and after 50 or so nodes the performance drops to a point at which using fabric switches becomes highly desirable. Arbitrated loops were used as a means of getting the advantages of FC at a lower cost. But with the cost of FC switches also dropping rapidly , the switched-fabric approach has become much more appealing.

Sections 4.4.2.1 through 4.4.2.3 describe some important concepts pertaining to FC-AL: loop initialization, loop arbitration, and different types of loops ”that is, public and private loops.

4.4.2.1 FC Loop Initialization

The Fibre Channel protocol for loop initialization follows a finite state machine. Loop initialization happens in the following cases:

• When the loop is first installed and started

• When a new device is plugged in

• When an existing device is removed or restarted

Loops are initialized by means of some special control frames . During initialization, the following activities occur:

• A loop master is elected for the duration until the next loop initialization is needed. One important function of the loop master is to play a role in assigning addresses to the various ports on the loop.

• Loop address assignment is accomplished. The address to which a device is assigned is important because the address value decides which device gets priority when multiple devices wish to transfer data simultaneously . Fabric switches have the highest priority; they can claim a particular address of their choice. Ports are allowed to claim an address they previously owned.

4.4.2.2 FC Loop Arbitration

When a port needs to communicate with another port, it must first arbitrate for loop ownership. To do this, a port sends a special frame called the ARB (short for "arbitration") primitive , which includes the loop address of the port seeking control of the loop. When the downstream port receives the ARB primitive, it simply forwards the primitive if it does not seek to do any communication. If the receiving port itself wishes to communicate, it compares the address in the ARB primitive with its own. If the port finds that it has a higher priority ”that is, its own address has a lower value ”it sends an ARB primitive with its own address. Otherwise it simply sends the ARB primitive that it received. At some point a port receives back its own ARB primitive. At that point it has control of the loop.

The port that won arbitration next sends an OPN (" open ") primitive targeted at the port with which it wishes to communicate. This primitive is simply passed on by intermediate ports until it arrives at the target port. The target port responds with a different ARB primitive, and once that is received by the initiating port, communication can begin. When the initiating port is done sending its communication, it sends a CLS ("close") primitive . However, the target port may continue sending frames to complete its communication, and the initiating port must be ready to receive these frames even though it already sent a CLS primitive. When the target port is done sending, it responds to the CLS primitive by sending a CLS primitive of its own. At this point the loop is ready for a new communication.

Fibre Channel specifies an optional algorithm to prevent lower-priority devices from getting starved because higher-priority devices continuously wish to communicate. This optional algorithm prevents a higher-priority device from contending for arbitration once it has successfully communicated until lower-priority devices have had a chance to communicate.

4.4.2.3 Public and Private Loops

A private loop is a Fibre Channel arbitrated loop that stands alone ”that is, is not connected to a fabric (which is described in Section 4.4.3). A public loop is an arbitrated loop that is connected to a fabric through a fabric switch. In this case, the fabric switch has to work hard to provide compatibility. Here are some examples:

• The fabric switch needs to ensure that addresses on the fabric and loop are unique and also map between three byte addresses on the fabric and a single byte address on the loop. This process is often referred to as address spoofing .

• The fabric switch needs to make devices on the fabric visible to devices on the loop as if they were simply present on the loop.

• Device discovery is essential for public and private communication. Fabric-aware Fibre Channel devices register themselves with the Simple Name Server (SNS, described in Section 4.4.3.1), but FC devices on a loop do not. It is the responsibility of the switch to probe each device in the FC loop and add its characteristics to the SNS for the benefit of the fabric-aware FC devices.

These are simple and not exhaustive examples of the problems involved in supporting a public loop.

4.4.3 Switched Fabric

In Fibre Channel switched-fabric topology, each device has a logical connection to any other device. Note the use of the qualifier logical in the previous sentence . It would be extremely difficult and costly to provide any-to-any connectivity via direct physical connections because for N devices, you would need N ² ports and physical connections. Hence each device is connected to a switch, and the switch implements logical connections among all the ports it has.

Figure 4.3 shows the simplest possible Fibre Channel switched-fabric topology. Multiple nodes (storage devices and computer systems) are connected to a Fibre Channel fabric switch. The switch is a high-speed device that can implement an any-to-any connection and handle multiple simultaneous connections. The switch also implements services such as Fabric Login, which is described in Section 4.4.3.6. Switches may be connected in a cascade or mesh fashion to build a more complex network. Very often a three- tier hierarchy is built with some FC-AL loops at the lowest level, some of which are connected via a lower-end switch and then in turn are connected via extremely high speed/high traffic “capable high-end switches.

Figure 4.3. Fibre Channel Switched-Fabric Topology

There may also be multiple switches that can be connected to each other. A fabric topology uses a 3-byte (24-bit) identifier to uniquely identify each device and theoretically provides any-to-any connectivity among up to 2 ²⁴ (approximately 15 million) devices. In practice, of course, SANs have a much lower number of devices deployed.

Because of the high number of potential devices, it is essential that Fibre Channel SANs offer services for address management as well as other services; otherwise it would be impossible to maintain SANs with manual management. Dynamic address management that is handled by the inherent infrastructure (as opposed to requiring manual intervention) is essential, given the large address space and the fact that the mapping between FC and upper-layer protocols such as SCSI is maintained by the nodes themselves.

A fabric topology is implemented with a fabric switch that is described in Section 4.4.3.5. Besides providing the any-to-any connectivity, the fabric switch provides services such as Simple Name Server, Registered State Change Notification (RSCN), Fabric Address Notification, IP-over-FC Broadcast Server, Principal Switch, and Fabric Login. Sections 4.4.3.1 through 4.4.3.6 briefly describe these services. Note that these services have a client/server model, and that although the fabric switch implements the server component, the client component needs to be implemented in devices such as HBAs and RAID controllers.

4.4.3.1 Simple Name Server

The Simple Name Server (SNS) is a database implemented within a fabric switch that tracks two things:

1. Device name and address information

2. Upper-layer protocols being supported

SNS allows an initiator of communication ”for example, a server HBA ”to query the name service and discover what storage devices are available. The basic idea here is to let the topology manage addresses by itself, which is highly desirable, given that switched-fabric SANs provide for a 3-byte address that can have up to 15 million unique addresses.

When a fabric-capable device initializes, it sends a login primitive (called FLOGI in FC terminology) to the well-known address 0xFFFFFE. The login request is filled with a zero value for address, indicating that an address is being requested .

Once a device has logged in to the fabric, it initiates another request (called a PLOGI primitive) to the Simple Name Server. The address for the Simple Name Server is fixed, always 0xFFFFFC. The device sends over information, such as its World Wide Name, port type, fabric address, upper-layer protocols (e.g., SCSI) that it supports, and other service parameters.

Some devices do not send all their information to the SNS. Some switch implementations send targeted probe requests to the device and update the appropriate field entries in the SNS data structures.

4.4.3.2 Registered State Change Notification

The Registered State Change Notification (RSCN) service allows a device to register for notifications when devices of interest join or leave the fabric. For example, a server could register for notifications that are triggered when a storage device joins or leaves the fabric. Thus, RSCN complements the SNS: Whereas SNS allows an FC entity (e.g., a host) to discover another entity (e.g., a storage unit), RSCN allows the host (FC entity) to register for notification of changes to FC entities.

RSCN packets may interrupt data transfer. RSCN frames are used by receivers to update their routing tables. Some vendors provide features in switches to suppress the transmission of RSCN packets widely. Although this suppression is good, the danger is that a device that really needs to see the RSCN may never see it, because of a bad configuration.

4.4.3.3 Fabric Address Notification

The basic idea here is to provide functionality for devices to ensure that their ongoing transactions are still valid. This functionality is typically useful when a loop initialization interrupts ongoing communication between two FC entities.

4.4.3.4 Broadcast Server

Because FC supports upper-layer protocols, it is essential that the needs of upper-layer protocols are met. One of the upper-layer protocols supported is Internet Protocol (IP). Because one of the secondary protocols supported in IP networks is Address Resolution Protocol, in which IP devices issue a broadcast to resolve addresses, it is essential that FC support similar functionality.

4.4.3.5 Principal Switch

When an FC SAN has multiple fabric switches, we need a protocol to select one of them as the principal switch. This principal switch acts as the switch that has the address 0xFFFFFE and ensures that the devices in the FC SAN have unique and nonconflicting addresses.

4.4.3.6 Fabric Login

Fabric-aware devices always attempt Fabric Login, by sending a special frame called an FLOGI frame to the well-known address 0xFFFFFE. The FLOGI primitive has a field for the address of the sending device, which can be set to zero to indicate that an address is being requested.

Fabric Login accomplishes some important functions, including

• Retrieval of information about fabricwide parameters ”for example, the class of services supported

• Address assignment

• Buffer credit initialization for flow control

---

Top