Fibre Channel

As previously stated, the FC network model includes some transport layer functionality. End-to-end connections are established via the PLOGI ELS command. Communication parameters may be updated during an active connection via the discover N_Port parameters (PDISC) ELS command. Segmentation and reassembly services are also supported. The following discussion considers only CoS 3 connections.

FC Operational Overview

PLOGI is mandatory for all N_Ports, and data transfer between N_Ports is not permitted until PLOGI completes. PLOGI may be performed explicitly or implicitly. Only explicit PLOGI is described in this book for the sake of clarity. PLOGI establishes an end-to-end connection between the participating N_Ports. PLOGI is accomplished with a single-frame request followed by a single-frame response. In switched FC environments, PLOGI accomplishes the following tasks:

• Provides the port worldwide name (PWWN) and node worldwide name (NWWN) of each N_Port to the peer N_Port

• Provides the operating characteristics of each N_Port to the peer N_Port

Like a TCP host, an FC node may open and maintain multiple PLOGI connections simultaneously. Unlike a TCP host, only a single PLOGI connection may be open between a pair of nodes at any point in time. This requires all ULPs to share the connection. However, ULPs may not share an Exchange. Similar to TCP connections, PLOGI connections are long-lived. However, keep-alives are not necessary because of the registered state change notification (RSCN) process discussed in Chapter 3, "Overview of Network Operating Principles." Another difference between TCP and FC is the manner in which ULP data is segmented. In the FC model, ULPs specify which PDU is to be transmitted in each sequence. If a PDU exceeds the maximum payload of an FC frame, the PDU is segmented, and each segment is mapped to an FC frame. FC segments a discrete chunk of ULP data into each sequence, so FC can reassemble the segments (frames of a sequence) into a discrete chunk for a ULP (such as FCP). So, receiving nodes reassemble related segments into a PDU before passing received data to a ULP. The segments of each PDU are tracked via the SEQ_CNT field in the FC header. In addition to the SEQ_CNT field, the Sequence Qualifier fields are required to track all data transmitted on a connection. Together, these fields provide equivalent data tracking functionality to the IP Source Address, IP Destination Address, TCP Source Port, TCP Destination Port, and TCP Sequence Number fields.

Two methods are available for segmentation and reassembly of ULP data: SEQ_CNT and relative offset. For each method, two modes are defined. The SEQ_CNT method may use a per-sequence counter (called normal mode) or a per-Exchange counter (called continuously increasing mode). The relative offset method may consider the value of the SEQ_CNT field in the FC header (called continuously increasing mode) or not (called random mode). If the SEQ_CNT method is used, frames received within a single sequence or a consecutive series of sequences are concatenated using the SEQ_CNT field in the FC header. If the relative offset method is used, frames received within a single sequence are concatenated using the Parameter field in the FC header. The method is specified per sequence by the sequence initiator via the Relative Offset Present bit of the F_CTL field in the FC header.

Note that the value of the Relative Offset Present bit must be the same in each frame within each sequence within an Exchange. When the Relative Offset Present bit is set to 1, relative offset is used. When the Relative Offset Present bit is set to 0, SEQ_CNT is used. FCP always uses the relative offset method. To use a particular method, both N_Ports must support that method. The supported methods are negotiated during PLOGI. The SEQ_CNT method must be supported by all N_Ports. The relative offset method is optional. The SEQ_CNT bit in the Common Service Parameters field in the PLOGI ELS indicates support for SEQ_CNT continuously increasing mode. The Continuously Increasing Relative Offset bit, the Random Relative Offset bit, and the Relative Offset By Info Category bit in the Common Service Parameters field in the PLOGI ELS indicate support for each mode of the relative offset method, and for applicability per information category.

Like TCP, FC supports multiple ULPs. So, FC requires a mechanism equivalent to TCP port numbers. This is provided via the Type field in the FC header, which identifies the ULP contained within each frame. Unlike TCP, FC uses the same Type code on the host (initiator) and storage (target) to identify the ULP. Because no more than one PLOGI connection can be open between a pair of nodes at any point in time, there is no need for unique connection identifiers between each pair of nodes. This contrasts with the TCP model, in which multiple TCP connections may be open simultaneously between a pair of nodes. Thus, there is no direct analog for the concept of TCP sockets in the FC model. The S_ID and D_ID fields in the FC header are sufficient to uniquely identify each PLOGI connection. If multiple ULPs are communicating between a pair of nodes, they must share the connection. This, too, contrasts the TCP model in which ULPs are not permitted to share TCP connections.

Multiple information categories are defined in FC. The most commonly used are Solicited Data, Unsolicited Data, Solicited Control, and Unsolicited Control. (Chapter 8, "OSI Session, Presentation, and Application Layers," discusses Solicited Data and Unsolicited Data.) FC permits the transfer of multiple information categories within a single sequence. The ability to mix information categories within a single sequence is negotiated during PLOGI. This is accomplished via the Categories Per Sequence sub-field of the Class 3 Service Parameters field of the PLOGI ELS.

The ability to support multiple simultaneous sequences and Exchanges can improve throughput significantly. However, each sequence within each Exchange consumes resources in the end nodes. Because every end node has limited resources, every end node has an upper limit to the number of simultaneous sequences and Exchanges it can support. During PLOGI, each node informs the peer node of its own limitations. The maximum number of concurrent sequences that are supported across all classes of service is indicated via the Total Concurrent Sequences sub-field of the Common Service Parameters field of the PLOGI ELS. The maximum number of concurrent sequences that are supported within Class 3 is indicated via the Concurrent Sequences sub-field of the Class 3 Service Parameters field of the PLOGI ELS. The maximum number of concurrent sequences that are supported within each Exchange is indicated via the Open Sequences Per Exchange sub-field of the Class 3 Service Parameters field of the PLOGI ELS.

FC Frame Formats

The PLOGI ELS and associated LS_ACC ELS use the exact same frame format as fabric login (FLOGI), which is discussed in Chapter 5, "OSI Physical and Data Link Layers," and diagrammed in Figure 5-25. Figure 5-25 is duplicated in this section as Figure 7-11 for the sake of convenience. Some fields have common meaning and applicability for FLOGI and PLOGI. Other fields have unique meaning and applicability for PLOGI. This section highlights the meaning of each field for PLOGI.

Figure 7-11. Data Field Format of an FC PLOGI/LS_ACC ELS Frame

A brief description of each field follows:

• LS command code 4 bytes long. It contains the 1-byte PLOGI command code (0x03) followed by 3 bytes of zeros when transmitted by an initiating N_Port. This field contains the 1-byte LS_ACC command code (0x02) followed by 3 bytes of zeros when transmitted by a responding N_Port.

• Common service parameters 16 bytes long. It contains parameters that affect network operation regardless of the CoS. Key parameters include the MTU, the offset mechanism for segmentation and reassembly, the maximum number of concurrent sequences across all classes of service, the SEQ_CNT policy, and the length of the PLOGI payload (116 or 256 bytes). Some parameters can be configured by the network administrator manually. If manually configured, only the values configured by the administrator will be advertised to the peer device.

• N_Port name 8 bytes long. It contains the PWWN of the N_Port.

• Node name/fabric name 8 bytes long. It contains the NWWN associated with the N_Port.

• Class 1/6, 2, 3, and 4 service parameters each 16 bytes long. They contain class-specific parameters that affect network operation. Key parameters relevant to Class 3 include indication of support for Class 3, CS_CTL preference, DiffServ, clock synchronization, the types of Exchange Error Policy supported, the maximum number of concurrent sequences within each class of service, the maximum number of concurrent sequences within each exchange, the class-specific MTU, and the ability to mix information categories within a single sequence. Some parameters can be configured by the network administrator manually. If manually configured, only the values configured by the administrator will be advertised to the peer device.

• Vendor version level 16 bytes long. It contains vendor-specific information.

• Services availability 8 bytes long. It is used only during FLOGI.

• Login extension data length 4 bytes long. It indicates the length of the Login Extension Data field. This field is valid only if the Common Service Parameters field indicates that the PLOGI payload is 256 bytes long. If present, this field must be set to 120.

• Login extension data 120 bytes long. It contains the vendor identity and other vendor-specific information. This field is valid only if the Common Service Parameters field indicates that the PLOGI payload is 256 bytes long.

• Clock synchronization QoS 8 bytes long. It contains operational parameters relevant to the fabric's ability to deliver clock synchronization data. It is used only if the Clock Synchronization service is supported by the switch. This field is valid only if the Common Service Parameters field indicates that the PLOGI payload is 256 bytes long.

The PDISC ELS and associated LS_ACC ELS use the exact same frame format as PLOGI. The meaning of each field is also identical. The LS Command Code field contains the PDISC command code (0x50). The PDISC ELS enables N_Ports to exchange operating characteristics during an active connection without affecting any sequences or exchanges that are currently open. This is useful when operating characteristics change for any reason. For the new operating characteristics to take affect, the connection must be terminated and re-established. Thus, PDISC is merely a notification mechanism.

FC Delivery Mechanisms

As previously stated, the FC model provides multiple sets of delivery mechanisms via multiple Classes of Service. All Classes of Service are implemented primarily at the data-link layer. This contrasts with the TCP/IP model, in which the choice of transport layer protocol determines which set of delivery mechanisms are supported. The majority of modern FC-SANs are configured to use Class 3. Class 3 delivery mechanisms are discussed in Chapter 5, "OSI Physical and Data-Link Layers," so this section discusses only the additional delivery mechanisms provided by PLOGI.

When a missing frame is detected, the behavior of the receiving node is determined by the Exchange Error Policy in effect for the Exchange in which the error is detected. The Exchange originator specifies the Exchange Error Policy on a per-Exchange basis via the Abort Sequence Condition bits of the F_CTL field of the FC header of the first frame of each new Exchange. The Exchange originator may specify any one of the Exchange Error Policies that are supported by the target node. Initiators discover which Exchange Error Policies are supported by each target node via the Error Policy Supported bits of the Class 3 Service Parameters field of the PLOGI ELS. See Chapter 5, "OSI Physical and Data-Link Layers," for a description of each Exchange Error Policy. Target nodes may support all three policies (Process, Discard Sequence, Discard Exchange) or only the two Discard policies. Regardless of the Exchange Error Policy in effect, detection of a missing frame is always reported to the ULP.

As discussed in Chapter 5, "OSI Physical and Data-Link Layers," the CS_CTL/Priority field of the FC header can be interpreted as CS_CTL or Priority. The CS_CTL interpretation enables the use of DiffServ QoS and frame preference. The Priority/Preemption interpretation enables the use of priority QoS and Class 1 or 6 connection preemption. Each node discovers the fabric's ability to support DiffServ QoS/frame preference and Priority QoS/connection preemption during FLOGI via the DiffServ QoS bit, Preference bit, and Priority/Preemption bit, respectively. All three of these bits are contained in the Class 3 Service Parameters field of the FLOGI ELS. Likewise, during FLOGI, each node informs the fabric of its own support for each of these features. For each feature supported by both the fabric and the node, the node is permitted to negotiate use of the feature with each peer node during PLOGI. This is accomplished via the same three bits of the Class 3 Service Parameters field of the PLOGI ELS.

Each node discovers the PMTU during FLOGI. However, a node might not be able to generate and accept frames as large as the PMTU. So, each node informs each peer node of its own MTU via the Receive Data Field Size bits of the Class 3 Service Parameters field of the PLOGI ELS. The lower of the PMTU and node MTU is used for all subsequent communication between each pair of nodes. Because PLOGI is required before any end-to-end communication occurs, fragmentation is precluded.

Comprehensive exploration of all aspects of the delivery mechanisms facilitated by PLOGI is outside the scope of this book. For more information about the end-to-end delivery mechanisms employed by FC, readers are encouraged to consult the ANSI T11 FC-FS and FC-LS specification series.

FC Connection Initialization

Class 3 service does not provide guaranteed delivery, so end-to-end buffering and acknowledgements are not required. This simplifies the connection establishment procedure. Following FLOGI, each N_Port performs PLOGI with the FC switch. This is required to facilitate Fibre Channel name server (FCNS) registration and subsequent FCNS queries as discussed in Chapter 3, "Overview of Network Operating Principles." Following PLOGI with the switch, each target N_Port waits for PLOGI requests from initiator N_Ports. Following PLOGI with the switch, each initiator N_Port performs PLOGI with each target N_Port discovered via the FCNS. For this reason, proper FCNS zoning must be implemented to avoid accidental target access, which could result in a breach of data security policy or data corruption (see Chapter 12, "Storage Network Security").

An initiator N_Port begins the PLOGI procedure by transmitting a PLOGI ELS frame to the FC address identifier (FCID) of a target N_Port. The FCID assigned to each target N_Port is discovered via the FCNS. Upon recognition of the PLOGI request, the target N_Port responds to the initiator N_Port by transmitting a PLOGI LS_ACC ELS. Upon recognition of the PLOGI LS_ACC, the PLOGI procedure is complete, and the N_Ports may exchange ULP data.

Note that FC does not implement a connection control block in the same manner as TCP. As previously mentioned, TCP creates a TCB per connection to maintain state for each connection. In FC, state is maintained per exchange rather than per connection. After PLOGI completes, the initiator creates an exchange status block (ESB), assigns an OX_ID to the first Exchange, and associates the OX_ID with the ESB. The ESB tracks the state of sequences within the Exchange. The initiator then creates a sequence status block (SSB), assigns a SEQ_ID to the first sequence, associates the SEQ_ID with the SSB, and associates the SSB with the ESB. The SSB tracks the state of frames within the sequence. The first frame of the first sequence of the first Exchange may then be transmitted by the initiator. Upon receipt of the first frame, the target N_Port creates an ESB and an SSB. The target N_Port then assigns an RX_ID to the Exchange and associates the RX_ID with the ESB. The value of the RX_ID is often the same as the value of the OX_ID. The target N_Port then associates the SEQ_ID received in the frame with the SSB and associates the SSB with the ESB. The target N_Port may then process the frame and pass the ULP data to the appropriate FC-4 protocol. This entire procedure is repeated for each new Exchange.