Basic SANs: Arbitrated Loops

The earliest SAN technology, which is still around today, is the Arbitrated Loop. Some vendors refer to this technology as Fibre Channel-Arbitrated Loop, or FC-AL for short. This is a simple physical ring topology (which can be wired as a star by using a hub) that bears some resemblance to Token-Ring technology. The word arbitrated indicates that a node on this SAN must contend for the network along with others connected to the ring. An arbitration process is used and, as with Token-Ring, only one node on the ring can exchange data with another node at the same time. The other nodes must wait to use the shared ring network media.

The Arbitrated Loop can be made up of up to 126 nodes, or 127 if you connect the loop to a fabric switch, which is discussed in the next section. The loop is self-addressing and uses a priority mechanism to access the SAN. Figure 11.2 shows an example of an Arbitrated Loop using a hub to centralize wiring of the SAN. In this figure note that the hub connects the transmitter of another node in the SAN, with the last transmitter connected back to the receiver on the first node plugged into the hub.

Figure 11.2. A hub is the simplest method for connecting members of an Arbitrated Loop SAN.

Unlike an Ethernet hub, in which a signal is broadcast to all other nodes on a segment, an Arbitrated Loop operates using a physical topology that resembles Token-Ring more than it does Ethernet. Because control and data frames must travel from one member of the loop to the next member physically wired to it, only two members of the loop can establish a link and exchange data at any single point in time. In the following sections you will learn how the Arbitrated Loop initializes itself to assign addressing and priority values to each member, and how members exchange data. Figure 11.3 shows the logical topology of how data flows around an Arbitrated Loop.

Figure 11.3. Data in an Arbitrated Loop uses a physical as well as logical loop topology.

Whether you string cables through your computer-room floor to connect one loop member to another, or use a hub to centralize wiring, the result is the same: a physical loop. This loop must be initialized when it is powered up, or when another member joins the loop. A means for gaining access to the network media is also needed so that each member of the loop can send or receive data.

Initializing the Loop

When a loop is powered up, or a new node joins the ring, a temporary loop master is elected and an initialization process takes place. The initialization process assigns addressing information to each member of the loop and sets the priorities of each member. The temporary loop master is chosen only for completing this initialization process, and it serves no function after that has been accomplished and the ring is functioning.

Each HBA, like an Ethernet card, has a hardware address burned into it by the manufacturer. This is a 64-bit number that is guaranteed (at least in newer equipment) to be unique throughout the world. The IEEE controls the range of numbers allocated to manufacturers. If an HBA has more than one port, each port also has its own unique hardware (or worldwide) address.

Note

Only newer SAN devices are likely to use worldwide names assigned by the IEEE. Older devices may have conflicting addresses, or require some configuration in order to assign this address.

These hardware addresses are used during the initialization process, and by some fabric switches. For the purpose of communicating on the ring, a separate address (24 bits) is used on the ring. This address is assigned or chosen during the initialization of the ring.

To begin the process, the temporary loop master must be chosen. A node begins the process by sending out a Loop Initialization Primitive (LIP). This LIP can be informative to the downstream neighbor if it does not yet know that the loop is being initialized. For example, a new device could have been added to the loop. Using this process, each node on the ring begins to transmit a LIP to its downstream partner in the ring.

This first LIP frame is called the Loop Initialization Select Master (LISM) frame. This frame contains information indicating its worldwide address and whether it is an NL_Port (node loop port) or an FL_Port (fabric loop port). The FL_Port can be used to connect an Arbitrated Loop to a larger fabric switched network.

If the LISM frame indicates that it was originated by an FL_Port, the receiving node stops transmitting its own LISM frame and instead repeats the FL_Port's frame. If it is just another NL_Port frame, the receiving node will compare its worldwide address with that of the received LISM frame. If the received frame has a higher priority (lower-numbered worldwide address), it is forwarded; otherwise, the receiver continues to forward its own LISM.

It takes only a few milliseconds before this technique notifies the node that either is connected to a fabric switch or has the lowest worldwide address on the loop that it is now elected the temporary loop master. Note that this designation can change as you add or remove hardware from the loop.

After the temporary loop master has been chosen (it has received back the LISM frame it sent out, so it knows that it must be the master of the loop), the temporary loop master sends out a frame informing the other members that it is assuming responsibilities as the loop master.

To finish the initialization process, it is necessary to assign values to the 24-bit IDs assigned to each member of the loop. This is accomplished by the temporary loop master sending around the loop four frames, each of which performs part of the process of assigning and recording the loop IDs of each node attached to the loop.

Note

The 24-bit address used as a port identifier is made up of two parts. The higher 2 bytes are assigned an identifier by a switch if the loop is attached to a switch. This is so that a switch can keep track of which nodes of an Arbitrated Loop are connected at what point. The lower 8 bits are used to assign one of 126 addresses to members of the loop (or 127 if a fabric attachment is used). This address, as a whole, is known as the AL_PA (Arbitrated Loop Physical Address). AL_PAs are used as sender and recipient addresses in Frame Relay communications.

Several of the next few initialization frames use a bitmap, in which each bit represents a numeric address that represents a byte address field. In other words, each bit in this sort of bitmap is used by the nodes that are part of the initialization process to represent an 8-bit (byte) field. In this manner, a single 256 range of bits can be used to represent a 256-byte address range. For example, bit 1 is used to indicate the first 8-byte address, bit 2 is used to indicate the second 8-byte address, and so on.

The first frame, called the Loop Initialization Fabric Address (LIFA), circles the loop and allows any member that remembers a switch assigned address to retain that address. This can easily occur when a new member has joined the loop and other nodes remember their previous addresses. Each node that remembers its previous address sets the bit in the bit map to indicate that the address has already been taken.

The second frame, called the Loop Initialization Previous Address (LIPA) frame, allows other nodes to claim an address if those nodes remember addresses assigned by some method other than by a switch. These nodes set the bit in the positional bit map to indicate that the address is already taken.

The third frame, called the Loop Initialization Hard Address (LIHA) frame, is used from older equipment. Such devices required that the administrator manually set an ID (worldwide address) for a device. Because these devices have a fixed address, this frame allows them to reserve it. Note that it may be necessary to change a hard address, and also that some newer equipment allows you to select the burned-in address or a manual address.

The last address selection frame is a catch-all frame called the Loop Initialization Soft Address (LISA) frame. Any node that does not yet have an address can select from those still available in the bit map. It is typical for servers and other important devices to select a higher-priority address, while other resources, such as disk arrays, select a lower-priority address.

After the temporary loop master has received this last frame, it can construct a map of all the nodes on the network as far as numeric addresses are concerned. The second matter to determine is to create a map showing the positions in the ring of those addresses. That is, the nodes around the ring do not have to have addresses that are in any order. The selection process lets each node, depending on circumstances, select an address.

So the Loop Initialization Report Position (LIRP) frame is next sent out to collect information about which addresses have been used, and in what order. Instead of a bitmap with each bit representing an address, this frame instead has a byte reserved for each address. As each node in the loop receives this frame, it fills in its address in the first available byte. Thus, when this frame arrives back at the temporary loop master, it has now an ordered listing of the topology of the loop.

The loop master then sends out a frame, called the Loop Initialization Loop Position (LILP) frame, that tells the other members of the loop about this physical location versus address information.

To finish the initialization process, the temporary loop master sends out a CLS (close) sequence. All nodes then enter an idle state and wait until a node in the loop wants to establish a connection with another node.

After this process has completed, it is necessary for a node on the Arbitrated Loop to gain access to the loop, log into a remote port, and then send or receive data.

Arbitrating for Loop Access

After the loop has started passing idles, a node on the loop can initiate a transfer of data to or from another node. This is accomplished by first logging into any ports that the node wants to establish a communications channel with. After the Port Login (PLOGI) sequence is completed, the two nodes can exchange information.

However, to gain access to the loop shared media (arbitrate for access), the node must send a frame around the loop requesting the access. This access is chosen by the priority set by the AL_PA. Similar to the method used to choose the temporary loop master, the frame circles the ring. If it arrives back at the node that sent it out onto the network, that node has permission to open a connection with another node and transfer one or more frames.

If another node that wants to begin a transmission receives this frame first, and notices that it has a higher priority than the incoming frame, the node will instead transmit its own. Again, if it receives this frame back, it has won the arbitration and can set up a session and transmit data to/from a Fibre Channel SANs device.

As you can see, as long as the bandwidth is sufficient (and Fibre Channel has progressed from speeds lower than 1Gbps to 10Gbps now), many devices can be accommodated in an Arbitrated Loop and satisfy many smaller clusters of servers, or single servers. If you have a much larger network, you'll probably need to upgrade to a fabric switched model so that you can add further capacity, and at the same time preserve your existing technology in Arbitrated Loops.