FCIP Operational Details


This section provides an in-depth exploration of the operational details of FCIP. This section complements and builds upon the FCIP overview provided in Chapter 4.

Architecture and Functional Models

TCP/IP is one of four technologies that enable geographically dispersed FC-SANs to be connected via non-FC network links (called SAN extension). The other three are SONET, asynchronous transfer mode (ATM), and transparent generic framing procedure (GFP-T). These four technologies are collectively called FC backbone networks. FC backbone networks serve as transparent virtual links between FC-SANs. The architecture and functional models for FC backbone networks are defined in the FC-BB specification series published by ANSI T11. The FC-BB-3 specification defines a model interface between FC and IP networks, and procedures for connection authentication and management, addressing, time synchronization, frame forwarding, error detection, and error recovery. The FC-BB-3 specification also mandates the protocol by which FCIP devices may be dynamically discovered. If dynamic discovery is supported, SLPv2 must be used. The FC-BB-3 specification requires all FC backbone networks to support Class F frames. Class 2, 3, and 4 frames may be optionally supported. Class 1 and 6 frames are not supported. FC primitive signals and primitive sequences are transmitted across only GFP-T links. The other three FC backbone networks do not forward FC primitives across non-FC links.

Additional specifications published by IETF define the details of FCIP functionality within the architecture defined in the FC-BB series. Generic procedures for encapsulating/de-encapsulating FC frames in TCP packets, mapping FC frame delimiters into 8-bit codes, and measuring unidirectional transit time between FC-BB devices are defined in RFC 3643. FCIP is defined in RFC 3821, which builds upon RFC 3643 by defining protocol-specific procedures for FC frame encapsulation and de-encapsulation, connection establishment and management, flow control, and tunnel security. Detailed procedures for optional dynamic discovery of FCIP devices using SLPv2 are defined in RFC 3822.

Note

The FC-BB specification series explicitly discusses FCIP. By contrast, iFCP is not explicitly discussed.


The FC backbone architecture provides WAN connectivity for FC-SANs as shown in Figure 8-32. The details of the inner workings of each FC-BB device are determined by the functional model of the FC-BB device. The functional model is determined by the type of non-FC network to which the FC-BB device connects. The FC-BB-3 specification defines two functional models for FC-BB devices that connect to IP networks: virtual E_Port (VE_Port) and B_Access. The VE_Port model is implemented by FC switches with integrated FCIP functionality. Currently, Cisco Systems is the only FC-SAN vendor that supports the VE_Port model. The B_Access model is implemented by FCIP devices that are external to FC switches. Devices that implement the B_Access functional model are called FCIP bridges. This can be confusing because bridges operate at OSI Layers 1 and 2, yet FCIP operates at OSI Layer 5. To clarify, FCIP bridges operate at OSI Layers 1 and 2 regarding FC-SAN facing functions, but they operate at OSI Layers 1 through 5 regarding WAN facing functions. The interface within an FCIP bridge to which an FC switch connects is called a B_Port (see Chapter 5). B_Ports support limited FC-SAN functionality.

Figure 8-32. FC Backbone Architecture


VE_Port Functional Model

An FCIP device that supports the VE_Port functional model has three conceptual components: the FC Interface, the IP Interface, and the FC-BB_IP Interface. The FC Interface implements FC-0 through FC-2 and interacts with the FC-SAN. The IP Interface implements TCP and IP and interacts with the IP network. The FC-BB_IP Interface resides between the other two interfaces and maps FC-2 functions to TCP functions. The FC-BB_IP Interface is composed of a switching element, an FC Entity, an FCIP Entity, a control and service module (CSM), and a platform management module (PMM). Figure 8-33 illustrates the relationships between the components of the VE_Port functional model.

Figure 8-33. VE_Port Functional Model-Component Relationships


F_Ports, E_Ports, and VE_Ports communicate through the Switching Element. Each FC Entity may contain one or more VE_Ports. Likewise, each FCIP Entity may contain one or more FCIP link end points (FCIP_LEPs). Each VE_Port is paired with exactly one FCIP_LEP. The four primary functions of a VE_Port are:

  • Receive FC frames from the Switching Element, generate a timestamp for each FC frame, and forward FC frames with timestamps to an FCIP_DE.

  • Receive FC frames and timestamps from an FCIP_DE, validate each FC frame's timestamp, and forward valid FC frames to the Switching Element.

  • Transmit Class F frames to the peer VE_Port.

  • Respond to Class F frames received from the peer VE_Port.

Each FC Entity is paired with exactly one FCIP Entity. Each FC-BB_IP Interface may contain one or more FC/FCIP Entity pairs. Each FCIP link terminates into an FCIP_LEP. Each FCIP_LEP may contain one or more FCIP data engines (FCIP_DEs). Each FCIP link may contain one or more TCP connections. Each TCP connection is paired with exactly one FCIP_DE. The six primary functions of an FCIP_DE are:

  • Receive FC frames and timestamps from a VE_Port via the FC Frame Receiver Portal (a conceptual port).

  • Encapsulate FC frames and timestamps in TCP packets.

  • Transmit TCP packets via the Encapsulated Frame Transmitter Portal (a conceptual port).

  • Receive TCP packets via the Encapsulated Frame Receiver Portal (a conceptual port).

  • De-encapsulate FC frames and timestamps from TCP packets.

  • Transmit FC frames and timestamps to a VE_Port via the FC Frame Transmitter Portal (a conceptual port).

Note

The terms FCIP tunnel and FCIP link are synonymous.


Figure 8-34 illustrates the FCIP_DE conceptual model.

Figure 8-34. FCIP_DE Conceptual Model


The CSM is responsible for establishing the first TCP connection in each FCIP link (FCIP link initialization). To facilitate this, the CSM listens on the FCIP WKP for incoming TCP connection requests. IANA has reserved TCP port number 3225 as the FCIP WKP. The SAN administrator may optionally configure the CSM to listen on a different TCP port. The CSM is also responsible for establishing additional TCP connections within existing FCIP Links. When a new FCIP link is requested, the CSM creates an FC/FCIP Entity pair. The FCIP Entity encapsulates FC frames without inspecting the FC frames. So, the FCIP Entity cannot determine the appropriate IP QoS setting based on the FC Header fields. Thus, the FC Entity determines the appropriate IP QoS policies for FC frames that are passed to the FCIP Entity. The FC Entity accesses IP QoS services via the CSM. The CSM is also responsible for TCP connection teardown, FCIP link dissolution, and FC/FCIP Entity pair deletion.

Whereas the FC Entity is responsible for generating a timestamp for each FC frame passed to the FCIP Entity, the PMM is responsible for ensuring that the timestamp is useful at the remote end of the FCIP link. This is accomplished by synchronizing the clocks of the FC switches at each end of an FCIP link. Two time services may be used for this purpose: FC Time Service or Simple Network Time Protocol (SNTP). SNTP is most commonly used. The total unidirectional transit time, including processing time for FCIP encapsulation and de-encapsulation, is defined as the FCIP transit time (FTT). Upon receipt of an FC frame and timestamp from an FCIP Entity, the FC Entity subtracts the value of the timestamp from the current value of FC switch's internal clock to derive the FTT. If the FTT exceeds half of the E_D_TOV, the frame is discarded. Otherwise, the frame is forwarded to the destination FC end node.

The PMM is also responsible for discovery of remote FCIP devices. An FC switch may be manually configured with the IP address, TCP port, and FCIP Entity Name of a remote FCIP device. Alternately, SLPv2 may be used for dynamic discovery of FCIP devices. The PMM interfaces with the SLPv2 service and makes discovered information available to the CSM. The PMM is also responsible for certain security functions (see Chapter 12, "Storage Network Security") and general housekeeping functions such as optional event and error logging.

After an FCIP link is established, the VE_Ports at each end of the link can communicate. VE_Port pairs communicate via a virtual ISL (VISL). Each FCIP link maps to a single VISL. The FCIP link logically resides under the VISL and serves the equivalent function of a physical cable between the FC switches. VE_Ports communicate over a VISL in the same manner that E_Ports communicate over a physical ISL. For example, VE_Ports use Class F frames (ELP, ESC, and others) to initialize and maintain a VISL. Likewise, FC frames are routed across a VISL in the same manner as a physical ISL. When an FC-SAN is connected to multiple remote FC-SANs, the FSFP protocol decides which VISL to use to reach a remote FC-SAN. After FC frames are encapsulated in TCP packets, IP routing protocols are responsible for forwarding the IP packets through the IP network. At the destination FC-SAN, the FSFP protocol makes the final forwarding decisions after FC frame de-encapsulation. Figure 8-35 illustrates the communication that occurs at each layer between FC switches that employ the VE_Port functional model. Note that the layers in Figure 8-35 do not correlate to the OSI layers. Note also that Figure 8-35 indicates which Classes of Service are permitted by the standards. In practice, only Classes 3 and F are used.

Figure 8-35. VE_Port Functional ModelLayered Communication


B_Access Functional Model

The B_Access functional model is very similar to the E_Port functional model. In most respects, the two models are identical. The primary differences are the replacement of the VE_Port with the B_Access port within the FC Entity and the use of a B_Port within the FC Interface to connect to an external FC switch. Figure 8-36 illustrates the relationships between the components of the B_Access functional model.

Figure 8-36. B_Access Functional ModelComponent Relationships


Communication between B_Access peers is similar to communication between VE_Port peers. The primary difference is the use of the Exchange B_Access Parameters (EBP) SW_ILS. As discussed in Chapter 5, the ELP SW_ILS is exchanged between the E_Port in the FC switch and the B_Port in the FC-BB bridge device. The parameters conveyed via ELP must then be conveyed across the WAN between FC-BB devices. The EBP SW_ILS performs this function. All other SW_ILS frames that are normally exchanged between FC switches over a physical ISL are exchanged between FC switches over the VISL provided by the FC-BB devices (see Figure 5-29 in Chapter 5). Figure 8-37 illustrates the communication that occurs at each layer between FC switches and FC_BB bridge devices that employ the B_Access functional model. Note that the layers in Figure 8-37 do not correlate to the OSI layers. Note also that Figure 8-37 indicates which Classes of Service are permitted by the standards. In practice, only Classes 3 and F are used.

Figure 8-37. B_Access Functional ModelLayered Communication


For more information about the FC backbone architecture and functional models, readers are encouraged to consult the ANSI T11 FC-BB-3 specification, and the IETF RFCs 3821 and 3822.

FCIP Addressing Scheme

FCIP does not provide host access to storage as does iSCSI and FCP. Therefore, the FCIP addressing scheme does not map to the SAM addressing scheme. That said, FCIP does use globally unique WWNs to distinguish each instance of an FCIP component from other instances and to facilitate reliable authentication. Such WWNs comply with the ANSI T11 WWN format. FCIP assigns WWNs as follows:

  • Each VE_Port is assigned a WWN. This is called the VE_Port_Name.

  • Each B_Access is assigned a WWN. This is called the B_Access_Name.

  • Each FC/FCIP Entity pair is assigned a WWN. This is called the FC/FCIP Entity identifier.

FCIP also uses WWNs that are assigned by FC. FC assigns WWNs as follows:

  • Each FC switch is assigned a WWN regardless of the presence of FCIP devices. This is called the Switch_Name.

  • Each FC-SAN is identified by the Switch_Name of the Principal Switch regardless of the presence of FCIP devices. This is called the Fabric_Name.

  • Each F_Port is assigned a WWN regardless of the presence of FCIP devices. This is called the F_Port_Name.

  • Each E_Port is assigned a WWN regardless of the presence of FCIP devices. This is called the E_Port_Name.

  • Each B_Port is assigned a WWN. This is called the B_Port_Name.

An FCIP_LEP that requests establishment of an FCIP Link is called an FCIP Link Originator. An FCIP_LEP that responds to such a request is called an FCIP Link Acceptor. In the VE_Port functional model, each FCIP Link Originator/Acceptor is identified by the following:

  • Switch_Name

  • FC/FCIP Entity identifier

  • VE_Port_Name

In the VE_Port functional model, each FCIP Link is identified by the following:

  • Switch_Name of the FCIP Link Originator

  • FC/FCIP Entity identifier of the FCIP Link Originator

  • VE_Port_Name of the FCIP Link Originator

  • Switch_Name of the FCIP Link Acceptor

In the B_Access functional model, each FCIP Link Originator/Acceptor is identified by the following:

  • Fabric_Name

  • FC/FCIP Entity identifier

  • B_Access_Name

In the B_Access functional model, each FCIP Link is identified by the following:

  • Fabric_Name of the FCIP Link Originator

  • FC/FCIP Entity identifier of the FCIP Link Originator

  • B_Access_Name of the FCIP Link Originator

  • Fabric_Name of the FCIP Link Acceptor

FCIP is very flexible regarding IP addressing. IP addresses may be assigned using one or more of the following methods:

  • A single IP address may be shared by all FCIP Entities within an FC switch or FC-BB bridge device

  • Each FCIP Entity within an FC switch or FC-BB bridge device may be assigned an IP address

  • Each FCIP_LEP within each FCIP Entity may be assigned an IP address

  • Each FCIP_DE within each FCIP_LEP may be assigned an IP address

Furthermore, FCIP devices at each end of an FCIP link are not required to implement IP addressing via the same method. For more information about FCIP addressing, readers are encouraged to consult the ANSI T11 FC-BB-3 specification.

FCIP Packet and Frame Formats

RFC 3643 defines the generic packet format to be used by all protocols that transport FC across IP networks. Figure 8-38 illustrates the generic FC Frame Encapsulation (FC-FE) format. Note that the entire FC frame, including frame delimiters, is encapsulated.

Figure 8-38. FC Frame Encapsulation Format


The encapsulation header is word-oriented, and an FC-FE word is four bytes. The FC-FE header contains many ones complement fields to augment the TCP checksum. Figure 8-39 illustrates the FC-FE header format.

Figure 8-39. FC-FE Header Format


A brief description of each field follows:

  • Protocol # This is 8 bits long. This field contains the IANA assigned protocol number of the encapsulating protocol. FCIP is protocol 1.

  • Version This is 8 bits long. This field indicates which version of FC-FE is being used. In essence, this field indicates the format of the packet. Currently, only version 1 is valid.

  • -Protocol # This is 8 bits long. This field contains the ones complement of the Protocol # field. This field is used to validate the value of the Protocol # field.

  • -Version This is 8 bits long. This field contains the ones complement of the Version field. This field is used to validate the value of the Version field.

  • Encapsulating Protocol Specific This is 64 bits (8 bytes) long. This field is used by the encapsulating protocol (such as FCIP) in a manner defined by the encapsulating protocol.

  • Flags This is 6 bits long. This field currently contains a single flag called the CRC Valid (CRCV) flag. Bits 0 through 4 are reserved and must be set to 0. The CRCV flag occupies bit 5. When the CRCV bit is set to 1, the CRC field is valid. When the CRCV bit is set to 0, the CRC field is ignored. FCIP always sets the CRCV bit to 0. This is because FCIP encapsulates the FC CRC field for end-to-end data integrity verification, and FCIP does not modify FC frames during encapsulation/de-encapsulation. Thus, the FC-FE CRC field is redundant.

  • Frame Length This is 10 bits long. This field indicates the total length of the FC-FE packet (FC-FE header through FC-FE EOF) expressed in 4-byte words.

  • -Flags This is 6 bits long. This field contains the ones complement of the Flags field. This field is used to validate the value of the Flags field.

  • -Frame Length This is 10 bits long. This field contains the ones complement of the Frame Length field. This field is used to validate the value of the Frame Length field.

  • Time Stamp (Seconds) This is 4 bytes long. This field may be set to 0 or contain the number of seconds that have passed since 0 hour on January 1, 1900. As discussed in the VE_Port functional model section of this chapter, the value of this field is generated by the FC Entity. The format of this field complies with SNTPv4.

  • Time Stamp (Second Fraction) This is 4 bytes long. This field may be set to 0 or contain a number of 200-picosecond intervals to granulate the value of the Time Stamp (Seconds) field. As discussed in the VE_Port functional model section of this chapter, the value of this field is generated by the FC Entity. The format of this field complies with SNTPv4.

  • CRC This is 4 bytes long. This field is valid only if the CRCV flag is set to 1. This field is ignored in FCIP implementations and must be set to 0.

RFC 3821 defines the format of the Encapsulating Protocol Specific field (Words 1 and 2) as implemented by FCIP. Figure 8-40 illustrates the FCIP format of the Encapsulating Protocol Specific field.

Figure 8-40. FCIP Format of Encapsulating Protocol Specific Field


A brief description of each field follows:

  • Word 1 This contains an identical copy of Word 0 in the FC-FE header.

  • Protocol Specific Flags (pFlags) This is 8 bits long. This field currently contains two flags called the Changed (Ch) flag and the special frame (SF) flag. The Ch flag occupies bit 0. Bits 1 through 6 are reserved and must be set to 0. The SF flag occupies bit 7. When the SF flag is set to 1, the FCIP packet contains an FCIP special frame (FSF) instead of an FC frame. When the SF flag is set to 0, the FCIP packet contains an FC frame. The Ch flag may be set to 1 only when the SF flag is set to 1. When the Ch flag is set to 1, the FSF has been changed by the FCIP Link Acceptor. When the Ch flag is set to 0, the FSF has not been changed by the FCIP Link Acceptor.

  • Reserved This is 8 bits long. It must be set to 0.

  • -pFlags This is 8 bits long. This field contains the ones complement of the pFlags field. This field is used to validate the value of the pFlags field.

  • -Reserved This is 8 bits long. It must be set to all ones.

The FC frame delimiters must be encapsulated because they serve several purposes rather than merely delimiting the start and end of each FC frame (see Chapter 5). However, FC frame delimiters are 8B/10B encoded words (40 bits long). Some of the 10-bit characters that compose FC SOF and EOF words have no 8-bit equivalent. Therefore, FC SOF and EOF words cannot be decoded into 32-bit words for transmission across an IP network. The solution to this problem is to represent each SOF and EOF word with an 8-bit code that can be encapsulated for transmission across an IP network. These codes are called Ordered Set Codes (OS-Codes). They are defined in the FC-BB-3 specification. The OS-Codes relevant to FC-FE are also listed in RFC 3643. Table 8-24 summarizes the OS-Codes that are relevant to FC-FE.

Table 8-24. OS-Codes Relevant to FC-FE

OS-Code

Frame Delimiter

Class of Service

0x28

SOFf

F

0x2D

SOFi2

2

0x35

SOFn2

2

0x2E

SOFi3

3

0x36

SOFn3

3

0x29

SOFi4

4

0x31

SOFn4

4

0x39

SOFc4

4

0x41

EOFn

F, 2, 3, 4

0x42

EOFt

F, 2, 3, 4

0x49

EOFni

F, 2, 3, 4

0x50

EOFa

F, 2, 3, 4

0x44

EOFrt

4

0x46

EOFdt

4

0x4E

EOFdti

4

0x4F

EOFrti

4


The format of the FC-FE SOF field is illustrated in figure 8-41. The FC-FE EOF field uses the same format (substituting EOF values for SOF values) and follows the same rules for sub-field interpretation.

Figure 8-41. FC-FE SOF Field Format


A brief description of each field follows:

  • First SOF This is 8 bits long. This field contains the OS-Code that maps to the FC SOF ordered set associated with the encapsulated FC frame.

  • Second SOF This is identical to the first SOF field.

  • First -SOF This is 8 bits long. This field contains the ones complement of the first SOF field.

  • Second -SOF This is identical to the first -SOF field.

When a TCP connection is established, the first packet transmitted by the FCIP Link Originator must be an FSF. The FCIP Link Acceptor may not transmit any packets until it receives the FSF. Upon receipt of the FSF, the FCIP Link Acceptor must transmit the FSF back to the FCIP Link Originator before transmitting any other packets. The FCIP Link Originator may not send additional packets until the echoed FSF is received. The FSF serves the following five purposes:

  • Conveys the originator's Switch_Name or Fabric_Name and FC/FCIP Entity Identifier to the acceptor

  • Discovers the Switch_Name or Fabric_Name associated with a known destination socket (if SLPv2 discovery is not used)

  • Conveys the intended destination Switch_Name or Fabric_Name to the acceptor (if SLPv2 discovery is used)

  • Declares the originator's operational parameters to the acceptor

  • Facilitates authentication of TCP connection requests

Note

Despite the FC-BB mandate that discovery be accomplished using SLPv2, RFC 3821 permits limited discovery using the FSF. However, RFC 3821 encourages the use of SLPv2 because SLPv2 supports discovery of more configuration parameters, is more extensible, and is more secure.


Figure 8-42 illustrates the FSF format.

Figure 8-42. FCIP Special Frame Format


A brief description of each field follows:

  • FC-FE header This occupies the first 7 words.

  • Reserved This is 16 bits long. It must be set to 0.

  • -Reserved This is 16 bits long. It must be set to all ones.

  • Source FC Fabric Entity WWN This is 64 bits (8 bytes) long. This field contains the Switch_Name or Fabric_Name of the device that originated the FSF.

  • Source FC/FCIP Entity Identifier This is 64 bits (8 bytes) long. This field contains the FC/FCIP Entity Identifier of the FC/FCIP Entity pair that originated the FSF.

  • Connection Nonce This is 64 bits (8 bytes) long. This field contains a 64-bit random number that uniquely identifies the TCP connection request. This field is used to authenticate additional TCP connection requests within an existing FCIP link (see Chapter 12).

  • Connection Usage Flags This is 8 bits long. This field indicates the Classes of Service that the FCIP link is intended to transport. In practice, only Classes 3 and F are transported. FCIP does not restrict the link usage to comply with the information conveyed by this field. This field is only used to facilitate collaborative control of IP QoS settings by the FC Entity and FCIP Entity.

  • Reserved This is 8 bits long and must be set to 0.

  • Connection Usage Code This is 16 bits long. This field is meant to contain FC-defined information related to IP QoS settings. RFC 3821 refers to the FC-BB-2 specification for detailed information about the use of this field. However, the FC-BB-2 and FC-BB-3 specifications both state that this field is reserved.

  • Destination FC Fabric Entity WWN This is 64 bits (8 bytes) long. This field contains the Switch_Name or Fabric_Name of the intended destination device. If the value of this field matches the receiving device's Switch_Name or Fabric_Name, the unchanged FSF is echoed back to the FCIP Link Originator with the Ch flag set to 0. If this field contains a WWN that does not match the receiving device's Switch_Name or Fabric_Name, the receiving device may correct the value of this field and echo the FSF back to the FCIP Link Originator with the Ch flag set to 1, or simply close the TCP connection without replying. Upon receipt of the changed FSF, the FCIP Link Originator must close the TCP connection. Having discovered the correct destination FC Fabric Entity WWN, the FCIP Link Originator may re-attempt to establish an FCIP link. If the FCIP Link Originator knows the socket of another FCIP device but does not know the FC Fabric Entity WWN of the device, the FCIP Link Originator may establish a TCP connection with that device and then send an FSF with this field set to 0. Upon receipt of the FSF, the receiving device may correct the value of this field and echo the FSF back to the FCIP Link Originator with the Ch flag set to 1, or simply close the TCP connection without replying. Upon receipt of the changed FSF, the FCIP Link Originator must close the TCP connection. Having discovered the destination FC Fabric Entity WWN, the FCIP Link Originator may re-attempt to establish an FCIP link.

  • Keep Alive Time-Out Value (K_A_TOV) This is 32 bits (4 bytes) long. This field indicates the maximum interval at which a Link Keep Alive (LKA) ELS should be sent in the absence of other traffic on an FCIP link. The default value for K_A_TOV is half of the E_D_TOV.

  • Reserved This is 16 bits long. It must be set to 0.

  • -Reserved This is 16 bits long. It must be set to all ones.

The FC-BB-3 specification defines the LKA ELS. Remember that a SW_ILS is an ELS that may be transmitted only between fabric elements. The LKA is exchanged only between VE_Port peers and B_Access peers. So, one might think that the LKA is a SW_ILS. However, the valid responses to an LKA are LS_ACC and LS_RJT (not SW_ACC and SW_RJT). Another key difference is that, unlike most SW_ILSs that must be transmitted in Class F Exchanges, the LKA may be transmitted using any Class of Service supported by the FCIP link. Thus, the FC-BB-3 specification refers to the LKA as an ELS instead of as an SW_ILS. The LKA serves two purposes: the LKA keeps an FCIP link active when no other traffic is traversing the link, and the LKA verifies link connectivity when no FCIP traffic is received for an abnormal period of time. In the case of multi-connection links, the LKA may be sent on each TCP connection. The number of unacknowledged LKAs that signify loss of connectivity is configurable, and the default value is 2. When a TCP connection failure is detected via LKA, the FC Entity may attempt to re-establish the connection (via the FCIP Entity) or redirect the affected flows to one of the surviving TCP connections within the FCIP link (assuming that another TCP connection is available). When an FCIP link failure is detected via LKA, the FC Entity must attempt to re-establish the FCIP link (via the FCIP entity). Support for the LKA is optional. If supported, at least one LKA should be sent per K_A_TOV during periods of inactivity, but LKAs may be sent more frequently. The LKA may be sent during periods of bidirectional activity, but the LKA serves no useful purpose during such periods. For implementations that support TCP keep-alives, the LKA might be unnecessary. However, the LKA is sent from one FC Entity to another. So, the LKA verifies connectivity across more of the end-to-end path than TCP keep-alives. For this reason, SAN administrators who wish to eliminate redundant keep-alive traffic should consider disabling the TCP keep-alive. Because the default K_A_TOV is lower than the typical TCP keep-alive interval, disabling the TCP keep-alive should have no negative side affects. If the TCP keep-alive is used, and the FCIP Entity detects a failed TCP connection, the FCIP Entity must notify the FC Entity and provide diagnostic information about the failure. The FC Entity decides whether to re-establish the TCP connection (via the FCIP Entity). An LKA may be rejected via the LS_RJT ELS, but receipt of an LS_RJT serves the purpose of keeping the link active and verifies connectivity to the peer FC Entity regardless of the Reject Reason Code. The FC-BB-3 specification does not define any Reject Reason Codes or Reason Explanations that are unique to the LKA. Receipt of an LS_ACC ELS signifies a successful response to an LKA. Both the LKA and its associated LS_ACC use the same frame format. Figure 8-43 illustrates the data field format of an LKA/LS_ACC ELS frame.

Figure 8-43. Data Field Format of an FC LKA/LS_ACC ELS Frame


A brief description of each field follows:

  • LS Command Code This is 8 bits long. This field contains the LKA command code (0x80) or the LS_ACC command code (0x02).

  • Unused These are each 8 bits long. They must be set to 0.

The FC-BB-3 specification defines the EBP SW_ILS. When using the B_Access functional model, the EBP complements the ELP. One B_Access portal transmits an EBP to another B_Access portal to exchange operating parameters similar to those exchanged via the ELP. The valid responses to an EBP are SW_ACC and SW_RJT. The SW_ACC contains the responder's operating parameters. No other frames may be transmitted before successful exchange of the EBP. The normal switch-to-switch extended link initialization procedure occurs after the EBP exchange (see Chapter 5). Both the EBP and its associated SW_ACC use the same frame format. Figure 8-44 illustrates the data field format of an EBP/SW_ACC SW_ILS frame.

Figure 8-44. Data Field Format of an FC EBP/SW_ACC SW_ILS Frame


A brief description of each field follows:

  • SW_ILS Command Code This is 4 bytes long. This field contains the EBP command code (0x28010000) when transmitted by a requestor. This field contains the SW_ACC command code (0x02000000) when transmitted by a responder.

  • R_A_TOV This is 4 bytes long. This field contains the sender's required R_A_TOV expressed in milliseconds.

  • E_D_TOV This is 4 bytes long. This field contains the sender's required E_D_TOV expressed in milliseconds.

  • K_A_TOV This is 4 bytes long. This field contains the sender's required K_A_TOV expressed in milliseconds.

  • Requester/Responder B_Access_Name This is 8 bytes long. In an EBP frame, this field contains the requester's B_Access_Name. In an SW_ACC frame, this field contains the responder's B_Access_Name.

  • Class F Service Parameters This is 16 bytes long. The format and use of this field are identical to its format and use in an ELP frame (see Chapter 5). This field contains various B_Access operating parameters. Key parameters include the class-specific MTU (ULP buffer size), the maximum number of concurrent Class F sequences, the maximum number of concurrent sequences within each exchange, and the number of end-to-end Credits (EE_Credits) supported. Some FC switches allow the network administrator to manually configure one or more of these values.

If a parameter mismatch occurs, the EBP command is rejected with an SW_RJT (see Chapter 5). The reason codes defined in the FC-SW-4 specification are used for EBP. However, the FC-BB-3 specification redefines the reason code explanations that are used for EBP. Table 8-25 summarizes the EBP reject reason code explanations. All reason code explanations excluded from Table 8-25 are reserved.

Table 8-25. EBP Reject Reason Code Explanations

Reason Code Explanation

Description

0x00

No Additional Explanation

0x01

Class F Service Parameter Error

0x02

Invalid B_Access_Name

0x03

K_A_TOV Mismatch

0x04

E_D_TOV Mismatch

0x05

R_A_TOV Mismatch


For more information about the FCIP packet formats and FC-BB frame formats, readers are encouraged to consult the IETF RFCs 3643 and 3821, and the ANSI T11 FC-BB-2, FC-BB-3, FC-LS, FC-SW-3, and FC-SW-4 specifications.

FCIP Session Establishment

FCIP link initialization cannot begin until the underlying technology at OSI Layers 1 and 2 (usually Ethernet) completes its initialization procedure. Next, an IP address must be assigned to each port that will support FCIP traffic. Most often, this is accomplished manually using static IP addresses. If IP security is implemented, IPsec initialization occurs next. The PMM then optionally initiates FCIP device discovery via SLPv2 and passes the discovered information to the CSM. Alternately, the CSM may retrieve peer FCIP device information from the local config file. The FCIP Link Originator may then establish the first TCP connection with each of its peer FCIP devices. After the TCP three-way handshake completes, the FCIP Link Originator transmits an FSF and then waits for the echoed FSF to arrive. At this point, the FCIP link is fully active.

The FC switch-to-switch extended link initialization procedure described in Chapter 5 then begins (excluding the transmission of FC primitives). To summarize that procedure:

1.

ELP is exchanged between VE_Ports or EBP is exchanged between B_Access portals.

2.

The VISL is reset to enable the operating parameters.

3.

ESC is exchanged between VE_Ports or E_Ports.

4.

Optional VE_Port or E_Port authentication occurs.

5.

PSS occurs.

6.

Domain_IDs are re-assigned if a Domain_ID conflict exists.

7.

Zone merge occurs.

8.

The FSPF protocol converges.

9.

An SW_RSCN is broadcast to announce the new VISL.

10.

Name Server updates are distributed.

11.

RSCNs are generated to announce the new Name Server records.

At this point, the two FC-SANs are merged into one FC-SAN. N_Ports can query their local Name Server to discover the new target devices in the remote segment of the merged FC-SAN. Additionally, Class 2, 3, and 4 frames can be exchanged between N_Ports across the new VISL.

FCIP Delivery Mechanisms

In FCIP environments, FC frame delivery failures can occur in any of three arenas: the IP network, the FC network, and the FC-BB_IP device. The delivery mechanisms of TCP, IP, and underlying technologies (such as Ethernet and SONET) are responsible for FCIP packet delivery in the IP network. For example, the TCP Sequence Number can be used to detect the loss of a TCP packet. Likewise, FC delivery mechanisms are responsible for FC frame delivery in the FC network. For example, the destination N_Port can detect a corrupt FC frame using the FC CRC. The handoff between these networks occurs within the FC-BB_IP device as previously described. During the handoff, certain problems can occur that result in dropped FC frames and/or TCP connection loss. These potential problems can be categorized as follows:

  • Time synchronization issues

  • FC frame corruption not detected by the TCP checksum

  • Flow-control issues

  • PDU boundary detection issues

If the FC Entities at each end of an FCIP link lose time synchronization, the timestamps delivered to the receiving FC Entity might result in dropped FC frames. Each incoming encapsulated FC frame must have an FTT equal to or less than half the E_D_TOV. Frames that do not meet this requirement must be discarded by the receiving FC Entity. When this happens, the destination N_Port is responsible for detecting the dropped frame and taking the appropriate action based on the Exchange Error Policy in effect for the impacted Exchange. For this reason, SAN administrators should ensure the time server infrastructure is redundant, and the relevant FC time-out values are set correctly to accommodate the FTT (taking jitter into account).

Note

Class F frames may be transmitted with a timestamp value of 0.


As previously discussed, the TCP checksum does not detect all bit errors. Therefore, the receiving FCIP Entity is required to validate each FC frame before forwarding it to the FC Entity. The receiving FCIP Entity must verify:

  • The SOF and EOF values are appropriate

  • The FC header is valid

  • The FC frame length is valid

Additionally, the FCIP Entity must verify that each packet received contains the proper content (for example, an FSF in the first packet received on each TCP connection). Invalid FC frames may be dropped by the FCIP Entity or forwarded to the FC Entity with an EOF that indicates that the frame is invalid. Forwarded invalid FC frames are dropped by the destination N_Port. FC is responsible for re-transmission in accordance with the Exchange Error Policy in affect for the affected Exchange.

The FC-BB-3 specification requires FC-BB_IP devices to coordinate the TCP/IP flow-control mechanisms with the FC flow-control mechanisms. Specifically, the FC Entity must cooperate with the FCIP Entity to manage flow control in both the IP network and the FC-SAN as if the two networks were one. In principle, this is accomplished by managing the number of BB_Credits advertised locally based on the size of the peer FCIP device's advertised TCP window. However, no specific methods are mandated or provided by the ANSI or IETF standards. So, the implementation choice is vendor-specific. If this requirement is not implemented properly, dropped FC frames or TCP packets can result.

Note

The challenges of mapping flow-control information between the FC and IP networks apply equally to iFCP implementations.


FCIP packets enter the TCP byte stream without the use of a message synchronization scheme. If an FCIP_DE loses message synchronization, it may take any one of the following actions:

  • Search the incoming TCP byte stream for a valid FCIP header and discard all bytes received until a valid FCIP header is found.

  • Close the TCP connection and notify the FC Entity.

  • Attempt to recover message synchronization for an implementation-specific period of time and then, if message synchronization is not recovered, close the TCP connection and notify the FC Entity.

If message synchronization cannot be recovered, and the TCP connection is closed, the FC Entity is responsible for re-establishing the TCP connection (via the FCIP Entity). Regardless of whether message synchronization is recovered, the affected N_Ports are responsible for detecting and re-transmitting all dropped FC frames in accordance with the Exchange Error Policy in effect for the affected Exchanges.

When the FCIP Entity detects an error, the FC Entity is notified via proprietary means. The FC Entity may notify the PMM via proprietary means for the purpose of error logging. The FC Entity must convert each error notification into a registered link incident report (RLIR). The RLIR is then forwarded to the domain controller of the switch. For certain errors, the FC Entity also forwards the RLIR to the management server of the switch. For more information about the FCIP delivery mechanisms and error reporting procedures, readers are encouraged to consult the IETF RFC 3821, and the ANSI T11 FC-BB-2, FC-BB-3, and FC-LS specifications.

FCIP Data Transfer Optimizations

Today, FCIP is used primarily for replicating data across long distances for the purpose of backup or disaster recovery. WAN circuits are often very expensive and represent a recurring cost. So, SAN administrators are always looking for ways to use their WAN circuits more efficiently. To that end, every major FCIP vendor supports one or more data transfer optimization features. The most common is data compression. Data compression works by replacing large, recognizable bit patterns with smaller bit patterns at the transmitting node. The receiving node then looks for the known smaller bit patterns in the incoming bit stream and replaces those patterns with the associated larger bit pattern. Data compression involves computational overhead, so hardware-based implementations are preferred because they offer dedicated computing power for compression operations. Cisco Systems supports hardware-based compression on some FCIP products and software-based compression on other FCIP products.

Another common optimization is the use of Ethernet jumbo frames. Using jumbo frames has the affect of reducing protocol overhead on the wire and reducing processing overhead on the end nodes. Note that the default Ethernet MTU of 1500 bytes is smaller than the largest FC frame size of 2148 bytes (including SOF and EOF). After adding the FC-FE header, TCP header and IP header, the largest encapsulated FC frame size is 2234 bytes. Thus, each FC frame must occupy more than one Ethernet frame. A common practice in FCIP environments is to increase the Ethernet MTU on the FCIP ports to 2300 bytes, which is large enough to accommodate a full-size FC frame and all associated FC-FE/TCP/IP encapsulation overhead (even if TCP options are used). Of course, this technique is fruitful only if jumbo frames are supported end-to-end, otherwise excessive fragmentation will occur (see Chapter 5). Cisco Systems supports jumbo frames on all FCIP products.

Note

MTU discovery during FLOGI does not account for FCIP links. FCIP links are completely transparent to N_ports. Also, the MTU of each FC-SAN that will be merged via FCIP must be the same.


As mentioned in the FCP section of this chapter, some of the newer optimizations being offered seek to accelerate SCSI operations. The most common technique is to spoof the FCP_XFER_RDY signal to eliminate excessive round trips across the WAN during SCSI write operations. Most, but not all, SAN extension vendors support some kind of FCP_XFER_RDY spoofing technique. Cisco Systems supports two FCP_XFER_RDY spoofing techniques on all FCIP products. One, for disk I/O, is called FCIP write acceleration (FCWA). The other, for tape I/O, is called FCIP tape acceleration (FCTA).




Storage Networking Protocol Fundamentals
Storage Networking Protocol Fundamentals (Vol 2)
ISBN: 1587051605
EAN: 2147483647
Year: 2007
Pages: 196
Authors: James Long

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net