Section 4.6. Fibre Channel Protocol

   

---

4.6 Fibre Channel Protocol

Fibre Channel is a set of standards developed by the American National Standards Institute (ANSI). Fibre Channel provides a high-performance serial link between host computers and storage units, as well as between storage units themselves . The standard allows for high-speed data transfers in a point-to-point or a loop network. Moreover, Fibre Channel provides all this with complete error checking.

The Fibre Channel standard defines five functional levels: FC-0 through FC-4. These are described in Sections 4.6.1 through 4.6.5. Note that in practical terms, most of the vendors provide for layers FC-0, FC-1, and FC-2 in silicon.

4.6.1 Layer FC-0

Layer FC-0 defines the physical characteristics of the interface and media. In particular, FC-0 defines the signaling, media, and receiver/transmitter specifications. FC-0 allows for several interfaces, to accommodate a variety of speeds and physical media. Examples of the physical media include copper , single-mode cable, and multimode cable. The transmission speed ranges from 12.5MB to 106.25MB per second. For those familiar with the ISO OSI 7 layer model for networking, ^[2] one can confidently say that FC-0 exists at the equivalent of layer 0 of the ISO OSI 7 layer model.

^[2] ISO stands for "International Organization for Standardization," and OSI stands for "Open Systems Interconnection."

4.6.2 Layer FC-1

Layer FC-1 defines the encoding and decoding of data, signal, and special characters , as well as error control. FC-1 also plays a role in link maintenance.

FC-1 uses an encoding scheme called 8B/10B . The scheme is designed to

• Allow easy synchronization

• Allow for enhanced error detection

• Allow for easy detection of control characters

• Simplify the design of the receiver/transmitter hardware

The 8B/10B encoding scheme transforms each 8 bits into two possibly different 10 bits of rendering. These 10 bits are presented in the form "Ann.m," where

• A = K to indicate command or D to indicate data

• nn = the decimal value of the last 5 bits of the byte

• m = the decimal value of the first 3 bits of the byte

The two possible different values arise from the fact that the specification chooses one of the two as the encoding to be transmitted on the basis of the immediate past history. The idea is to ensure a minimum number of state transitions (between 1 and 0) and therefore to ensure good transmission efficiency. This immediate past history is called running disparity .

As stated already, all data is encoded with 10 bits. Some of the unused 10-bit characters (from a data point of view) are used for frame delineation and signaling, including signaling to indicate that a port is ready to receive data, as well as other types of signaling. The emphasis is on error detection and correction during transmission. FC always transmits data using groups of 4 bytes called transmission words .

4.6.3 Layer FC-2

Layer FC-2 defines how data is transferred from one node to the next , in essence defining the transport mechanism. To state this differently, the FC-2 layer assembles frames , defines classes of service, and defines services for fabric or port login. Conceptually, FC-2 can be thought to be at the equivalent of the Media Access Control (MAC) layer of the ISO OSI 7 layer model.

The FC-2 layer defines

• The Fibre Channel transmission hierarchy, consisting of ordered set, frame, sequence, and exchange

• Fibre Channel flow control

• FC-2 protocols

• FC-2 classes of service

These entities are described in Sections 4.6.3.1 through 4.6.3.7. Before we consider them in more detail, it will be useful to understand how the various building blocks of the transmission hierarchy are put together.

Data is exchanged in Fibre Channel via an entity called a frame . A frame is the Fibre Channel equivalent of a TCP/IP packet. Frames are built of ordered sets and data characters. Multiple frames are grouped together to make a sequence , and multiple sequences together constitute an exchange . This relationship is shown in Figure 4.4.

Figure 4.4. Fibre Channel Transmission Hierarchy

Now that the overall view of the transmission hierarchy is clear, in Sections 4.6.3.1 through 4.6.3.4 we'll look at the individual components ”that is, ordered sets, frames, sequences, and exchanges.

4.6.3.1 FC Ordered Set

Ordered sets are simply 4-byte transmission sequences that represent special characters or special link signals. Examples of ordered sets include

• Frame delimiters SOF (start of frame) and EOF (end of frame) , which are analogous to the Ethernet packets SOF and EOF in functionality. Unlike Ethernet, in Fibre Channel multiple SOF or EOF ordered sets are defined because FC-1 uses an encoding scheme that can have multiple representations for each character transmitted.

• Two primitive signals to indicate port status:

  1. Idle indicates that a port is ready for transmission and reception .

  2. Receiver Ready indicates that an interface buffer (on an interconnect device) is ready for reception of data.

• A primitive sequence that is simply an ordered set transmitted continuously to denote special conditions at a port. These special conditions include

• Not Operational ( NOS ), which is used only in point-to-point and fabric topologies (i.e., not in arbitrated loops ), to indicate a link failure or error.

• Offline ( OLS ), which is transmitted when a port is initializing or when it receives a NOS primitive; that is, a port transmits an OLS as the response to a NOS primitive.

• Link Reset ( LR ), which is used to indicate that a link reset is desired.

• Link Reset Response ( LS ), which is used to indicate that the LR primitive has been received and acted upon.

4.6.3.2 FC Frame

Just as the basic building block for Internet Protocol (IP) is the IP packet, a frame is the basic building block for Fibre Channel. In general, there are three kinds of frames:

1. Link control frames to send link commands

2. Link data frames to send data needed for link control

3. Device data frames that carry data for higher-level protocols ”for example, data read off a disk

Figure 4.5 shows the Fibre Channel frame header. The frame is designed to be able to transmit 2,048 bytes of data and an optional 64-byte header. The large size of the frame means that a large amount of data can be transmitted rapidly with very little overhead (about 1.5 percent). On the other hand, another node that needs to transmit will have to wait until the large frame completes, implying a higher latency. Compare this with ATM (Asynchronous Transfer Mode), in which the frame size is 53 bytes with a protocol overhead of about 10 percent providing for a lower latency, but potentially higher transmission time at the same bit rate.

Figure 4.5. Fibre Channel Frame Header

Every frame begins and ends with a special delimiter , just as in other network protocols. These are the SOF and EOF delimiters, respectively. Every frame also has a header. The header has several purposes. One is to provide source and destination addresses to facilitate switching. Another is to carry information to control the link, including controlling the transfer of data.

The other fields in the Fibre Channel frame header are described as follows :

• The Destination_Id field is used to route the frame. In the point-to-point and loop topologies the routing may be trivial, but not in a switched-fabric topology. The Source_Id field is useful for reporting errors and avoiding routing loops.

• The R_CTL and Type fields are jointly used to distinguish various different FC-4 frames when they arrive at the destination. That is, together these fields tell whether a frame that has just arrived contains SCSI data, IP data, or other data. The values for the Type field are summarized in Table 4.3.

• The R_CTL field is used to indicate what the contents of the frame are. These could be data or link control frames and, in the case of link control frames, could consist of solicited or unsolicited frames.

• The F_CTL field is used to denote frame information such as first sequence or last sequence.

• The DF_CTL field identifies the presence or absence of optional headers.

• The SEQ_Id and SEQ_CNT fields uniquely identify a sequence count within an exchange (see Section 4.6.3.3).

• The OX_Id (originator exchange identifier) field is used to associate a frame with a specific exchange at the originating port.

• The RX_Id ( responder exchange identifier) field is used to associate a frame with a specific exchange at the responding port.

• The Relative Offset field identifies the relative displacement of the first byte of the payload from the base address.

Table 4.3. Values for the Type Field in Fibre Channel Frames

Value	Description
0x00	Basic link service
0x01	Fibre Channel port ”extended link service
0x02	Memory port
0x03	Memory initialization
0x04	IEEE 802.2
0x05	Internet Protocol
0x06	IPI (Intelligent Peripheral Interface) slave
0x07	IPI master
0x08	SCSI initiator
0x09	SCSI target
0x0A	HIPPI
0x0B	Single-Byte Command Code Sets (SBCCS)
0x0C	Futurebus
0x10	Reserved
0xD0	Vendor unique
0x FO	Direct channel

4.6.3.3 FC Sequence

A sequence is simply a series of frames transmitted from one port to another. To facilitate error recovery, each frame contains a unique sequence count. The error recovery is done by a higher-level protocol, typically at the FC-4 layer. Note that all the frames in a sequence are transmitted in one direction (and are not bidirectional). Figure 4.4 shows how frames, sequences, and exchanges are related .

4.6.3.4 FC Exchange

Multiple sequences make up an exchange. The sequences are a mixture of sequences in each direction; that is, even though a particular sequence is always unidirectional, the sequences that make up an exchange will flow in either direction. In each exchange, only one sequence can be active at any given moment. However, because different exchanges can be active at the same time, different sequences constituting those different exchanges can also be active at the same time.

Each Exchange performs a single function ”for example, implements a SCSI command such as Read.

4.6.3.5 FC Flow Control

Fibre Channel end nodes communicate directly with each other and do not make session connections with intervening nodes. They are unaware of the fabric switch and hub. Of course, fabric switches and hubs do exchange flow control packets with these devices.

A flow control mechanism is required to ensure that the sending port does not send frames faster than the receiver can take care of them. Fibre Channel ports have buffers to store the frames temporarily and process them later, processing being defined as either forwarding the frame to another port or passing the frame to an upper-layer protocol. The flow control scheme used by FC is similar to the sliding-window protocol used by TCP/IP. A window representing the number of frames that can be sent, but not acknowledged, is negotiated. This negotiation is binding and cannot be changed. For every frame that is sent, the window size is effectively decreased by one. For every frame that is acknowledged , the window size is effectively incremented by one. Flow control can be implemented in one of two ways: end-to-end and buffer-to-buffer. These are not mutually exclusive, and they are both required.

End-to-end flow control is between two endpoints: a data source such as a server, and a data sink such as a storage unit. End-to-end flow control always occurs between two N ports (there may be intervening nodes between them). The two N ports perform a login with each other, during which each port establishes how many buffers the other port has reserved for it. This amount is called the buffer credit . The sender can send at most this many frames. The receiver sends an ACK (positive acknowledgment) frame for each frame successfully received and processed , and the sender, upon receipt of the ACK frame, may increment the credit count by one for each ACK frame received. The receiver can also acknowledge successful receipt of multiple frames or an entire sequence, and the sender needs to increment the number of frames it can now send without waiting for acknowledgments.

Buffer-to-buffer flow control occurs between two adjacent nodes that either are intermediate nodes or are between an end node and an intermediate node. Thus, buffer-to-buffer flow control occurs either between N ports or between an F port and an N port. As already explained, the ports exchange values that indicate the number of buffers reserved by each node. These can be different; for example, one port may reserve two buffers while the other reserves four buffers. Frame receipt is acknowledged by means of a Receiver Ready frame rather than an ACK frame in end-to-end flow control.

4.6.3.6 FC-2 Protocols

Fibre Channel standards define protocols for managing data transmission and link control. Additional standards are also defined as needed to cater to higher-level protocols defined in FC-4. The protocols include the following:

• Fabric Login protocol , which defines the exchange of parameters between a port and the fabric. Fabric Login is described in more detail in Section 4.4.3.6.

• Port Login protocol , which requires that, no matter what the topology (point to point, arbitrary loop, or fabric switch), two ports need to perform a mutual login before they can communicate with each other. The mutual login is accomplished with a special frame called the PLOGI frame. Port Login accomplishes some important functionality, including the following:

  • The ability to learn about the N port to which the login is being performed. An example of the knowledge gained is the class of services that the N port supports.

  • Buffer credit initialization for end-to-end flow control. Note that for a direct connection, end-to-end flow control is the same as buffer-to-buffer flow control.

• Data Transfer protocol , which defines how upper-layer protocol (defined in FC-4) data is transferred via the flow control schemes defined in Section 4.6.3.5.

• Arbitrated Loop protocol , which defines loop initialization and arbitration.

4.6.3.7 FC-2 Service Classes

Fibre Channel was designed to cater to a wide variety of data transmission needs. The various services defined differ in one or more of the following characteristics:

• Orientation of the service connection ”that is, like TCP or connectionless, similar to UDP?

• Whether or not multicast is supported

• Whether the service provides notification of delivery and nondelivery

• Whether the service guarantees delivery of frames in the same order they were sent

• The nature of special services provided ”for example, reservation of bandwidth per connection if the service is connection oriented

• The nature of flow control mechanisms

To provide for a wide variety of transmission needs, the following different classes of service are defined:

• Class 1 service defines a dedicated connection, similar to a TCP/IP connection. Just as in TCP, Class 1 guarantees that frames are delivered in the same sequence as they were sent. Class 1 service is useful for transmitting large amounts of data where the time spent to set up the connection is negligible compared to the amount of time required to transmit the data.

• Class 2 defines a connectionless (datagram-like) service in which frames can potentially be delivered out of sequence (implying that a higher-level protocol must take care of receiving frames out of order and resequencing them). As in networking protocols, Class 2 service makes sense when the amount of data to be transmitted is rather small and the overhead of setting up a connection is comparable to the overhead of the actual data transmission itself. The receiver of a Class 2 frame is expected to send an acknowledgment upon successful receipt of the frame.

• Class 3 is also a connectionless service, similar to Class 2. The main difference is that the successful receipt of frames is not acknowledged. This is analogous to an IP datagram, where the transmission is sometimes referred to as "send and pray."

• Class 4 , also called Intermix , is an optional service. In this service, Class 1 frames are guaranteed a certain transmission bandwidth, and the remaining bandwidth, if any, is used for Class 2 and Class 3 frames.

• Class 6 is a unidirectional connection-oriented service that provides multicast capabilities. (Class 5 is a reserved class.)

Table 4.4 summarizes the information about FC service classes.

Note that most vendors support all of Classes 1, 2, and 3 in their implementations . To minimize silicon costs, some vendors support only connectionless service (Classes 2 and 3).

Table 4.4. Fibre Channel Service Classes

	Class 1	Class 2	Class 3	Class 4	Class 6
Nature of service	Connection	Connectionless	Connectionless	Connection	Connection
Multicast support	No	No	Yes	No	Yes
Delivery confirmation	No	No	Yes	No	Yes
Bandwidth used	Full	No guarantee	No guarantee	Partial	No guarantee
Flow control used	End-to-end	End-to-end	Buffer-to-buffer	Buffer-to-buffer	End-to-end
Frame delivery	Guaranteed to be in order	May be out of order	May be out of order	Guaranteed to be in order	Guaranteed to be in order

4.6.4 Layer FC-3

Layer FC-3 defines common services, including services for management and common transport. Whereas FC-3 is common across all ports that a node may have, the FC-0, FC-1, FC-2, and FC-4 layers are on a per-port basis. This arrangement supports the ability of different ports to have different configurations in use, as Figure 4.6 illustrates. For example, one port may be transporting SCSI data while another port is transporting ATM data.

Figure 4.6. Fibre Channel Layers

Further, note that whereas the bottom three layers ”FC-0, FC-1, and FC-2 ”are on a per-port basis, the FC-3 layer is on per-node basis. The FC-4 upper layer is also on a per-port basis. Figure 4.6 shows a node with four ports and two upper-layer protocols ”namely, SCSI and IP. If there were more upper-layer protocols, there would be more FC-4 boxes for the respective mapping functionality.

Some examples of common functionality that can be implemented at the FC-3 layer include

• Trunking or striping , in which parallel links and ports are " collapsed " to provide a thicker, fatter pipe between two nodes.

• Multicast , in which a single transmission can be delivered to multiple ports. Multicast is implemented via a registration service in the fabric switch by which nodes can register or deregister for a multicast group . The notion of multicast here is similar to the notion of multicast in Internet Protocol (IP). Note that a multicast could be to all ports on the fabric; that is, a multicast could also be a broadcast.

• Hunting , in which multiple ports have the same alias. This is similar to a company acquiring multiple telephone lines, all with the same number, and when a consumer dials the company number, the consumer is simply directed to one of the free lines. The obvious advantage is that the chances of reaching a free port are higher with hunting implemented.

Brocade is one example of a vendor that has implemented trunking, although Brocade prefers to call it Inter Switch Link (ISL) Trunking. The name is intended to convey the fact that Brocade promotes its high-end switches as being extremely high-speed switches that are intended to form the backbone of a switched fabric and are primarily intended to connect to other switches rather than to servers and storage devices. Note that other vendors may also have implemented trunking.

4.6.5 Layer FC-4

Layer FC-4 defines mapping of upper-level protocols to Fibre Channel. FC-4 defines mappings for the following protocols, and more will probably follow:

• SSCI

• Internet Protocol (IP)

• Intelligent Peripheral Interface (IPI)

• High-Performance Parallel Interface (HIPPI)

• IEEE 802.2

• SBCCS (Single-Byte Command Code Sets)

• ATM Adaptation Layer for computer data (AAL5)

• Link Encapsulation (FC-LE)

Note that a single Fibre Channel link can simultaneously carry multiple upper-layer protocols.

---

Top