6.2 Native IP Storage Protocols

A native IP protocol is one that maps a unique IP address to an individual networked device. In the earlier discussion on FCIP tunneling, you learned that IP addresses are assigned to the FCIP tunnel gateways but not to individual storage devices whose traffic passes through the IP tunnel. By contrast, both the Internet Fibre Channel Protocol (iFCP) and Internet SCSI (iSCSI) assign individual IP addresses to each storage participant. In this way, traffic from any device can leverage the capabilities of IP routing and TCP session control through the network, and you can use traditional IP network management tools to monitor conversations between initiators and targets.

The fact that there are two protocols in this category reflects the transitional nature of SAN plumbing. iFCP makes no mention of iSCSI and instead focuses on mapping Fibre Channel end devices to an IP transport. This lets you build SANs without Fibre Channel fabrics and overcome the problematic FCIP tunneling issues. Moreover, iSCSI makes no mention of Fibre Channel at all and instead focuses on building pristine IP SANs. iFCP therefore provides a migration path from Fibre Channel to IP SANs, whereas iSCSI attempts to create IP SANs from the ground up.

Because both iFCP and iSCSI are based on traditional TCP/IP networking, standardization is under the jurisdiction of the IETF. Standardization is not an end point but rather is an ongoing process as protocols are developed and augmented by auxiliary services. Both iFCP and iSCSI rely on IP routing protocols and TCP session control to ensure in-order delivery of data and packet-level recovery in the event of network congestion. The IP and TCP mechanisms therefore provide the foundation for transporting data reliably over IP network infrastructures.

6.2.1 Internet Fibre Channel Protocol

The Internet Fibre Channel Protocol (iFCP) is a gateway-to-gateway protocol for providing Fibre Channel fabric services to Fibre Channel end devices over a TCP/IP network. iFCP combines the fault isolation properties of autonomous regions with the scalability of the IP network. Although iFCP allows attachment to existing Fibre Channel fabric switches, an iFCP gateway also allows direct attachment of Fibre Channel end devices. As a result, you can create IP-based SANs that accommodate Fibre Channel storage and hosts but do not require Fibre Channel fabrics or fabric management.

As illustrated in Figure 6-10, an iFCP gateway has a dual personality. A Fibre Channel N_Port attached to an iFCP gateway is served by a standard F_Port interface on the gateway. This implies that the gateway can support fabric services, such as fabric login, SNS registration and queries, and State Change Notifications. Between the Fibre Channel device domain and the IP domain, iFCP must translate between Fibre Channel behavior and IP behavior. At the iFCP layer, a Fibre Channel device's 24-bit fabric address is mapped to a unique 32-bit IP address, providing native IP addressing for individual Fibre Channel initiators and targets. In place of the Fibre Channel lower-layer transport (FC-2), iFCP substitutes TCP/IP for reliable transmission over the IP network.

Figure 6-10. A simple IP SAN based on the iFCP gateway protocol

After Fibre Channel-originated data has been converted to IP format, it can be sent anywhere within the IP routed network. Discrete IP mapping thus enables convergence of SAN, LAN, and WAN storage applications using existing Fibre Channel assets. iFCP also enables use of mainstream Gigabit Ethernet switches to build the SAN core, an attractive option for customers who already have established acquisition and support channels for Gigabit Ethernet director and departmental switches.

A unique contribution of iFCP is the ability it gives you to create gateway regions that provide communications between multiple sets of Fibre Channel end devices and yet isolate potential disruptions within each region. Because iFCP gateways do not pass Fibre Channel switch-to-switch protocols from site to site, you avoid the need to create an extended fabric. Disruptive behavior within a local SAN is isolated to that gateway region and is not, as would be the case with FCIP, propagated throughout the network. The ability to preserve the autonomy of each local site and yet provide connectivity to other sites contributes to the stability of the IP SAN and restricts fabric-inspired disruptions.

Discrete mapping of individual Fibre Channel devices to IP addresses also enables any-to-any IP routing, as shown in Figure 6-11. Fibre Channel-originated storage conversations are not restricted to IP tunnels and can be routed to any desired destination in the IP network.

Figure 6-11. iFCP any-to-any communication across an IP routed network

This strategy provides flexibility in device deployment for wide area, metropolitan, and local IP SAN configurations and enables advanced disaster recovery implementations such as data replication between multiple sites.

Compared with simple FCIP tunneling, iFCP actively intervenes in the transportation of frame traffic between N_PORTs. The iFCP gateway must constantly manage Fibre Channel transactions and intercept port login requests when the boundary between Fibre Channel and IP domains is crossed. When an E_Port is attached to a Fibre Channel switch, the iFCP gateway must emulate a fabric while terminating the E_Port connection at the local site. In the network shown in Figure 6-11, for example, the Fibre Channel switch would see its locally attached iFCP gateway as a terminal point in the local fabric. All the Fibre Channel devices at the far sides of the cloud would appear to be attached directly to the local iFCP gateway. The iFCP gateway thus presents itself as a large virtual fabric switch with potentially hundreds or thousands of directly attached devices, although in reality those devices may be dispersed throughout an IP network.

The complexity of iFCP compared with FCIP tunneling has encouraged Fibre Channel switch vendors to select FCIP for SAN extension. In the process, however, the vulnerabilities of stretched E_Port connections and distributed fabrics are perpetuated. The engineering challenge for iFCP gateway design is to perform complex functions at wire speed while still abiding by both Fibre Channel and IP standards.

A native IP solution for Fibre Channel-originated traffic may actually extend the value of Fibre Channel storage assets. After Fibre Channel data has been converted to iFCP, it can benefit from established QoS and security services originally created for IP messaging. These enhancements are not yet available in Fibre Channel switch products but are readily available in IP network equipment. Consequently, the protocol conversion provided by iFCP amplifies the capabilities customers can leverage for the investment they have already made in Fibre Channel storage and hosts.

6.2.2 Internet SCSI (iSCSI)

The development of the iSCSI protocol fulfills customer requests dating back to the initial marketing of SANs. The ability to create storage networks with familiar and supportable Ethernet and IP technology was not feasible without gigabit performance and an IP protocol optimized for block storage data. Now that SANs have proven their end-user value and Gigabit Ethernet is readily available, development of iSCSI has received backing by all major storage and storage solutions providers.

The iSCSI protocol standard omits any reference to Fibre Channel because iSCSI replaces both Fibre Channel fabrics and Fibre Channel device interfaces with a native IP protocol. Whereas iFCP is intended to provide a migration path from Fibre Channel to IP SANs, iSCSI has no migration component. Both initiators and targets are iSCSI, with standard IP routing and Ethernet providing the link between them.

One of the fundamental contradictions that iSCSI has had to address is the issue of network intelligence. In Fibre Channel SANs (as well as iFCP gateways), the network interconnect must supply intelligence in the form of automatic address allocation, login, device discovery, and state change notification. This intelligence is required because storage targets are passive nodes in the network, playing a role that is analogous to their passive position on a SCSI daisy chain. IP networking, by contrast, assumes intelligence in the end nodes and little intelligence in the network itself. Specifically, current-generation IP network equipment may not be storage-aware. Because iSCSI cannot assume storage intelligence in the Gigabit Ethernet switch or IP router complex, an iSCSI device itself must have alternative means to deal with storage-specific phenomena. Some of the proactive functionality may be in the end device, whereas some may be supplied by additional boxes (such as SNS servers) attached to the network.

As with the FCP layer in Fibre Channel, iSCSI's primary task is to encapsulate SCSI commands, status, and data. At the core of an iSCSI packet, the traditional CDB (command descriptor block) performs the real work of reading or writing blocks of data to disk or tape. As shown in Figure 6-12, the SCSI CDB is carried in an iSCSI protocol data unit (PDU) in the same way it is carried by the Fibre Channel FCP layer. The iSCSI PDU contains other information to monitor the status of transactions between initiators and targets and in turn rides upon the TCP session control and IP routing layers.

Figure 6-12. The iSCSI protocol stack

One or more TCP connections are established to support an iSCSI session between initiator and target. More TCP sessions enable more concurrent transactions to be performed, thus optimizing utilization of the protocol processing and the link.

For routing purposes, each iSCSI device has a unique IP address. If the device is moved from one part of the network to another, its IP address might necessarily change. Consistent identity of a device regardless of its network address must therefore be supplied with something comparable to a Fibre Channel World-Wide Name (WWN). For iSCSI, as many as 255 bytes can be used to assign a unique identity to an iSCSI node. This potentially very long iSCSI name allows for both WWN-type identity and human-readable names that might specify location, device type, or departmental ownership.

In Fibre Channel fabrics, the first act of an attached device is to log in to the fabric switch and receive a Fibre Channel address. iSCSI also provides a login service, but instead of being directed to the network switch, the login is performed to another iSCSI device. The network IP address has already been assigned, either through manual configuration or via a DHCP (Dynamic Host Configuration Protocol) server. The iSCSI login sequence is used to negotiate any variable parameters between an initiator and a target and may invoke a security routine to authenticate allowable connectivity. If the connection is successful, the target issues a login acceptance to the initiator; otherwise, the login is rejected and the connection broken.

After login is completed, the iSCSI session enters into normal SCSI transactions. If multiple connections for that session have been established, individual command/response pairs must flow over the same TCP connection. This connection allegiance ensures that specific read or write commands are fulfilled without the additional overhead of monitoring multiple connections to see whether a particular request is completed. A SCSI write, for example, would be performed over a single connection until all data was transmitted. Unrelated transactions, however, could simultaneously be issued on their own connections during the same session.

As shown in Figure 6-13, an iSCSI transaction between initiator and target is very similar to an FCP transaction.

Figure 6-13. Execution of an iSCSI write operation

iSCSI protocol data units are used to send commands, status, and data, with Ready to Transmit (R2T) PDUs fulfilling the role of upper-layer SCSI flow control between target and initiator. In the SCSI write operation depicted in Figure 6-13, the R2Ts are issued by the target device to solicit data as receive buffers become available. At the completion of the write, the target issues status and sense, indicating a successful transaction.

iSCSI sessions and their associated TCP connections normally remain open, awaiting additional SCSI commands from the upper-layer applications. An initiator, for example, typically has assigned disk resources in the SAN and rarely breaks connection with them unless rebooted. In an iSCSI environment, however, initiator activity may require more TCP connections for some transactions than others, and so allowance is made for selectively logging out of previously established connections. In some instances, such as downing a resource for maintenance, it may be desirable to log out of a session and all connections completely. The iSCSI logout command supplies reason codes for terminating sessions or connections within a session, in the latter case using the connection ID (CID) to specify which connection should be taken down. In the event of a connection error, the logout command can be issued on an alternative connection (or a newly established one) to clean up the problematic TCP connection.

As the iSCSI protocol has worked its way through the standardization process, security of storage data over IP networks has been a central concern. Early standards drafts provided login processes for negotiating a common security methodology including public/private key exchanges and Kerberos. An attempt to mandate inclusion of more sophisticated IPSec encryption, however, met opposition by vendors who feared that burdening every iSCSI adapter card with the cost of encryption ASICs would undermine iSCSI adoption. (IPSec is discussed in Section 6.5.) In the end, customers will determine where and at what price they need the safeguards of data encryption. Top of the line iSCSI offerings may include on-board IPSec capabilities, whereas other iSCSI devices may depend on third-party security functionality provided by IP routers or dedicated firewall products.

The two IP-based technologies that are facilitating iSCSI for SANs are TCP off-load engines and 10Gbps Ethernet. TOEs are highly attractive for any TCP/IP processing and have a much broader market appeal than strictly storage-specific applications. By reducing host CPU overhead, TOEs enable iSCSI adapters to provide performance and utilization comparable to Fibre Channel HBAs. Similarly, 10Gbps Ethernet is being driven by the broader IP networking market, but it is especially suited for high-volume IP storage requirements. In addition to 10Gbps Ethernet interswitch links for building large IP SAN core infrastructures, 10Gbps Ethernet interfaces on storage arrays will enable simultaneous access for much higher populations of servers than traditional SANs. This capability helps to amortize the cost of storage access without sacrificing performance.