5.5 Arbitrated Loop Hubs

Primarily because of the successful marketing tactics of fabric switch vendors, loop hubs are less commonly deployed for SANs than are higher-priced switches. Loop hubs, however, are quite adequate for many moderate-performance shared storage applications and can be combined with fabric switches to provide a more economical solution.

Hubs are available from a number of vendors in various port configurations, interface types, and levels of management. Unlike other Fibre Channel devices, a hub is a passive participant in the SAN topology. Except for special management functions that may be provided by a vendor, a hub normally has no port address. So although all loop traffic passes through the hub, no NL_Port speaks directly to the hub itself. Finding a means to maintain this unintrusive role and yet engineer advanced management features into hub technology is a challenge that hub vendors have assumed.

5.5.1 Star Topology for Arbitrated Loop

As discussed in Chapter 4, you can wire arbitrated loops into a ring by connecting the transmit lead of one device to the receive lead of another and extending this scheme until all devices are physically configured into a loop. This technique eliminates the cost of a concentrating hub, but it exposes the topology to considerable risk. If a connection fails anywhere along the ring, the entire loop goes down. Troubleshooting is complicated by the fact that there is no central point where all cabling is brought together, and, because the loop is down, no in-band diagnostics (if available) can be used. In addition, introducing an NL_Port to the configuration (or removing it) requires downing the loop for the duration of the change. Bringing down a network for an add, move, or change usually is not tolerated outside test lab environments.

As illustrated in Figure 5-9, loop hubs simplify the cable plant by concentrating physical links at a central location, and they minimize disruption by providing bypass circuitry at each port. Each link in the physical loop is brought into the hub, and that creates a star cabling configuration while maintaining the circular data path through all devices. Port bypass circuitry automatically routes around any hub port that loses Fibre Channel signaling, and that prevents a broken cable or disabled NL_Port from disrupting loop operations.

Figure 5-9. Wiring concentration with an arbitrated loop hub

Hubs are usually co-located with storage arrays or servers in 19-inch equipment racks, and that offers a further convenience for verifying status and cabling within the enclosure. Large storage arrays may have multiple JBODs or RAIDs and multiple loop hubs configured into separate or redundant loops within a single 19-inch enclosure.

5.5.2 Hub Architecture

Hub design varies from vendor to vendor, but all hubs incorporate basic features specific to arbitrated loop. A hub embodies the loop topology by completing circuits through each port and then joining the transmit of the last port to the receive of the first. When cabling from an NL_Port is plugged in to a hub port, the port must, at minimum, be able to recover valid Fibre Channel clock. If the signaling is too fast or too slow, the port will remain in bypass mode. Depending on the sophistication of the loop hub, an insertion decision will be made on the basis of the received Fibre Channel signal or a combination of valid signal and valid ordered sets via protocol decode. For most unmanaged loop hubs, if the attached NL_Port is providing valid signal, the device is allowed to insert into the loop.

As shown in Figure 5-10, a port in bypass mode shunts the bit stream it has received from its upstream neighbor directly to its immediate downstream neighbor. If a device is removed from port 4, for example, the bypass circuitry will route traffic from port 3 to port 5. Port 4's transmitter, however, may still be active. Some vendor implementations turn off the transmitter as long as the port is in bypass mode. The transmitter is then enabled only when the hub port receives valid signal from a newly attached device. Other designs leave the transmitter on so that an attached device will immediately see valid Fibre Channel signal from the hub and thus insert without delay.

Figure 5-10. Internal hub architecture with port bypass circuitry

Auto-bypass circuitry allows the loop topology to self-configure when devices are added or removed. When servers or storage arrays are taken off line or reenabled, the circumference of the loop circuit automatically contracts or expands through the associated ports. At the link protocol level, loop initialization handles the automatic reconfiguration of port addresses to accommodate newly attached devices. The combination of hub auto-bypass and insertion with arbitrated loop's self-configuring addressing algorithm offloads tedious administration tasks from day-to-day storage network operations and facilitates changes to the topology with minimal disruption to data traffic.

Loop hubs may offer one or more light-emitting diodes (LEDs) per port to display port status. Typically, two LEDs are provided: a green LED to indicate a link connection, and an amber LED to indicate the current bypass mode. Depending on the vendor's specification, the combination of green and amber LED states can be used to display a number of port conditions, listed in Table 5-1. Some implementations use a single, multicolored LED per port, thereby reducing the number of diagnostic display states available. Port LEDs give the operator an at-a-glance status of a device's connection state, and by simplifying troubleshooting they help reduce down time.

Table 5-1. Hub Port LED State Table

Green	Amber	Port State
Off	Off	No device attached
On	Off	Device attached and loop inserted
On	On	Device attached and bypassed
Off	On	Bad GBIC, port bypassed
Blinking	Blinking	Maintenance mode via hub management

Hub architecture is not dictated by specific Fibre Channel standards except for the implied parameters established by arbitrated loop documents (FC-AL , FC-AL-2, and so on) and physical layer FC-0 standards. As long as vendors do not violate arbitrated loop and Fibre Channel physical layer provisions, they are free to engineer variations in port density, port type, signal processing, and management features to enhance functionality. Consequently, various hub products are available that are engineered to different marketing requirements, from simple entry-level unmanaged hubs to hubs with advanced diagnostics on-board.

Port density on a single unit may vary from as few as 5 ports to as many as 32. The most common density is from 6 to 12 ports per unit. This range is adequate for most loop implementations, and by cascading hubs together you can build higher loop populations. Loop hubs are typically 1-U high units, and that allows, for example, as many as 48 ports to be supplied in 4 slots of a 19-inch rack. With some models you can configure higher densities, but usually only by providing ports on both back and front of the hub chassis, thus complicating the cabling scheme.

Port types also vary from vendor to vendor. Fixed copper ports can be found on low-cost, entry-level loop hubs, although others in the same class offer GBIC-based ports. Fixed copper ports are not a highly desirable feature, because they are prone to high EMI emissions and jitter. Cross-talk, in which signaling from one port affects its neighbor, is also a concern with copper interfaces. Additionally, the lower price of fixed copper ports is offset by the extra cost of media interface adapters (MIAs) if you wish to use fiber-optic cabling from the hub to a device. Most mid-range hubs employ GBIC-based ports because of the flexibility GBICs offer for mixing copper, shortwave, and longwave media. This design allows you to provision ports as needed and to change from one media type to another without swapping out the hub itself. GBIC-based hubs may be marketed "without media" that is, without the additional cost of the GBICs themselves or may be bundled with a specific configuration of optical or copper GBICs (or both) preinstalled. Because most fabric switches also use GBICs, this hub architecture simplifies parts sparing for storage networks based on both fabrics and loops.

5.5.3 Unmanaged Hubs

Unmanaged hubs can be used for small, single-vendor environments that are less susceptible to the dynamics of larger, more complex SAN installations. If only a single server and one or two disk arrays are involved, there is less exposure to prolonged down time if a cable or other component fails. And if a single solutions provider has supplied the server, HBA, disk arrays, hub, GBICs, and cabling, there is less demand for the kind of higher-level SAN diagnostics required by a multivendor configuration.

Unmanaged hubs are a simple, low-cost, entry-level interconnection solution. They typically provide port bypass circuitry, based on valid signaling alone and port LEDs to display insertion or bypass status. If an attached device is unplugged or powered off, an unmanaged hub will auto-bypass the port and light the appropriate port LEDs. Having no intelligence or ordered set recognition circuitry, an unmanaged hub cannot respond to protocol violations or conditions, such as a port streaming LIP(F8), that would bring the loop down. The selection of an unmanaged hub therefore implies an assumed risk. The trade-off for lower cost and simplified installation is the longer mean time to repair that might follow a loop failure.

Some loop-based SAN configurations are deployed with redundant loops to provide an alternative path if the primary loop fails. The lack of visibility to the hub and loop status via hub management is balanced against the security offered by redundant paths. Such implementations are, by and large, lower risk, because the hubs and other interconnection components rarely fail. The worst case scenario for redundant, unmanaged loops would be the quiet failure of the backup loop, followed eventually by the failure of the primary loop. Without hub management to report the backup loop failure, the redundancy would no longer be in force, and the failure of the primary loop would bring the system down. Troubleshooting would then be complicated by the fact that what appeared to be a single occurrence system failure was actually the result of two separate events on two separate loop topologies.

Unmanaged hubs are a logical choice for low-cost storage network solutions in which economical servers and small JBODs are used to meet budget restraints. These entry-level systems bring SAN capability within reach of small business and departmental applications, and they create the initial infrastructure that can evolve to support more sophisticated requirements.

5.5.4 Managed Hubs

Managed hubs introduce intelligence into the SAN interconnection. The degree of intelligence and management capability varies from product to product and is normally reflected in price. At the low end of the managed hub offering, basic hub status and port controls are available via Web browser, Telnet, or SNMP (Simple Network Management Protocol) management software. At the high end, advanced diagnostics and proactive management features are available on a variety of management platforms, including HP OpenView and Java-based applications.

Support for management functionality in arbitrated loop hubs has two basic components. First, at the hardware level, additional on-board circuitry is needed to monitor power supply, fan, temperature, and port status. These functions are the minimum requirement for hub management, although some implementations concentrate on port status alone. Managed hubs with advanced capabilities also provide more extensive diagnostic circuits to monitor the state of the loop, protocol activity, and more comprehensive port and hub status. Second, for these circuits to be useful, the hub must be able to report to and accept commands from an external management workstation. This is typically accomplished via an Ethernet port on the hub, over which SNMP queries and commands are sent from an NT or UNIX console. The application software used to manage a hub is provided by the hub vendor, either as a stand-alone program or as a utility that can be launched from more comprehensive SAN management applications.

Managed hub products that support SNMP or Web browsers can be managed from anywhere in an IP routed network, as shown in Figure 5-11. This is a distinct advantage for customer support organizations and network command centers, as well as resellers or VARs that have support contracts with remote customers. SNMP is preferable to browser-based management because multiple hubs can be managed from a single workstation. Browser-based management typically speaks to only one IP address, and therefore to one hub, at a time. Chapter 10 discusses SNMP and other management protocols and strategies in more detail.

Figure 5-11. Managed SAN with remote SNMP console

Most vendors provide an event log for automatic logging of hub and loop events. The event log may be queried from the hub using Telnet or SNMP, or it may be replicated on the management workstation for a permanent record of activity. An event log is a useful diagnostic tool, especially for troubleshooting intermittent problems that may occur during unattended operation of the management workstation.

The graphical interfaces supplied with managed hubs vary in capability and platform support from vendor to vendor. At the low end, browser-based management may provide screens for basic port configuration and status but no active updates for visual cues on hub status. At the high end, NT or Java-based applications provide a "back of box" or portside depiction of the hub, with color-coded cues and legends for hub and port status. Pop-up alerts may be provided for significant events, as well as SNMP traps to send notification to another management platform. Depending on your management philosophy, a SAN management workstation may be co-located with the SAN or incorporated into a network operations center where LAN and WAN management is performed. Consequently, managed hub applications may provide additional interfaces to integrate into HP OpenView, Tivoli, or other comprehensive management platforms.