4.2 Disk Device Files in a Switched Fabric, Fibre Channel SAN

     

The problem we have here is that when we first experience a Fibre Channel SAN, most of us think that the Fibre Channel card is in some way akin to a SCSI card. I can understand that because at the end of our SCSI card are our disks and the end of our Fibre Channel card are our disks. What we need to remember is that Fibre Channel is a link technology that supports other protocols running on top of it. In relation to disks, this means the SCSI-2 protocol is running over Fibre Channel. This is the one of the beauties of Fibre Channel, it gives us the best of both worlds ”the distances of LANs and the ease-of-use of SCSI. In some ways, I consider fibre channel as a mile-long SCSI cable (I know it can be longer than that it was a joke). In Chapter 23, we discuss Fibre Channel in more detail. Here, we are going to concentrate on deciphering device files for Fibre Channel attached disks.

The most important thing to remember here is that Fibre Channel is supporting SCSI-2 protocols, so we will use SCSI-2 addressing to reference our disks. This means that the Instance number used in the device file for a disk is the Instance number of the ext_bus device to which we are connected. Let's take an example of a small SAN with two disk arrays attached to two interconnected switches:

The two switches are connected together, and each HP server has a single connection to each switch. The two disk arrays have been configured with two LUNs (LUN0 = 4GB and LUN1 = 2GB) and three LUNs (LUN0 = 0GB, LUN1 = 10GB and LUN200 = 5GB), respectively. Here, we have the output from ioscan show the ext_bus interfaces for node hp1 :

 

 root@hp1[]  ioscan -fnC ext_bus  Class     I  H/W Path      Driver   S/W State   H/W Type     Description ======================================================================== ext_bus   0  0/0/1/0       c720     CLAIMED     INTERFACE    SCSI C896 Ultra Wide LVD ext_bus   1  0/0/1/1       c720     CLAIMED     INTERFACE    SCSI C896 Ultra Wide Single-Ended ext_bus   2  0/0/2/0       c720     CLAIMED     INTERFACE    SCSI C87x Fast Wide Single-Ended ext_bus   3  0/0/2/1       c720     CLAIMED     INTERFACE    SCSI C87x Ultra Wide Single-Ended  ext_bus  10  0/2/0/0.1.1.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus  12  0/2/0/0.1.1.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface  ext_bus  14  0/2/0/0.2.1.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  15  0/2/0/0.2.1.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus  17  0/2/0/0.2.1.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface  ext_bus  18  0/2/0/0.2.2.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  20  0/2/0/0.2.2.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus  21  0/2/0/0.2.2.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface  ext_bus  11  0/4/0/0.1.1.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus  13  0/4/0/0.1.1.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface  ext_bus  16  0/4/0/0.2.1.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  19  0/4/0/0.2.1.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus  22  0/4/0/0.2.1.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface  ext_bus   4  0/4/0/0.2.2.0.0     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  23  0/4/0/0.2.2.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface  ext_bus   5  0/4/0/0.2.2.255.0   fcpdev   CLAIMED     INTERFACE    FCP Device Interface root@hp1[] 

You can see that I have underlined the ext_bus interfaces that are acting as our Array Interface ; these are this interfaces used by the SCSI protocol to communicate with LUNs configured on the disk array. The entry described as Device Interface is simply a reference for the disk array itself; it plays no part in the addressing of individual LUNs. I suppose we need to start by explaining a little about the hardware path itself. Let's take one of the addresses as an example:

 

 0/2/0/0.1.1.0.0 

The format of the hardware path can be broken into three major components :

 

 <HBA hardware path><Port ID><SCSI address> 

Let's take each part in turn :

<HBA hardware path> : An HBA in the world of Fibre Channel can be thought of as an interface card: HBA = Host Bus Adapter. In this example, the hardware path is

 

 0/2/0/0 

<Port ID> : Officially this is known as the N_Port ID . For our discussion, we can simplify it to just Port ID . The Port ID identifies the end-point in our SAN closest to our disks (LUNs). No matter how many switches we have traversed to get there, the Port ID is effectively the point at which I leave the SAN and start talking to my disk device. The Port ID is a 24-bit address assigned by a switch in our SAN. The 24-bit address is broken into three components:

  • A Domain (switch number) = 1

  • An Area (physical port on the switch) = 1

  • A Port (the FC-AL Loop ID hence always 0 in a switched fabric SAN)

These are the next three components in our hardware path:

 

 1.1.0 

A quick word of caution. In some (older) SANs, you may see the Area in our example as being 17. The firmware in a switch has a setting known as Core Switch PID Format . When turned off, we would add 16 to the physical port number to give the Area = 17 , taking the example above. When Core Switch PID Format is turned on, we simply use the physical port number on the switch. Most new switches come with this firmware setting turned on . A SAN will fragment (switches don't talk to each other) if switches use different settings.

IMPORTANT

Be careful if you ask your SAN administrator to change this firmware setting on your switches. If changed, HP-UX will see a whole new set of ext_bus interfaces all with the new Area number as part of the address. This means that a whole new set of disk device files associated with these new interfaces. If the disks are LVM disks, we will need to vgexport and vgimport entire volume groups in order to use the new disk device files. You have been warned !


<SCSI address> : The last component in the address of my ext_bus interface is known as the Virtual SCSI Bus (VSB) address. As we mentioned previously, Fibre Channel supports SCSI-2 protocols. Trying to visualize what this last component in the address is doing, I visualize this part of the address as the end of the SCSI cable. A Virtual SCSI Bus (visualize the end of SCSI cable) can support up to 128 devices. So at the end of this particular cable could be up to 128 LUNs. This will become important later.

Our example address of 0/2/0/0.1.1.0.0 can be seen in Figure 4-1 as the line near the top of the figure. The other colored lines relate to the hardware address of an ext_bus as follows in Table 4-1.

Figure 4-1. A sample switched fabric SAN
graphics/04fig01.jpg

Table 4-1. Addresses of SCSI Interface

Color

H/W Path

Instance Number

Orange

0/2/0/0.1.1.0.0

10

Lime Green

0/2/0/0.2.1.0.0

14

Red

0/2/0/0.2.2.0.0

18

Burgundy/Violet

0/4/0/0.1.2.0.0

11

Navy Blue

0/4/0/0.2.1.0.0

16

Yellow/Gold

0/4/0/0.2.2.0.0

4


The actual Fibre Channel cards themselves have their own Instance numbers :

 

 root@hp1[]  ioscan -fnkC fc  Class     I  H/W Path  Driver S/W State   H/W Type     Description ================================================================= fc        1  0/2/0/0   td   CLAIMED     INTERFACE    HP Tachyon TL/TS Fibre Channel Mass graphics/ccc.gif Storage Adapter                       /dev/td1 fc        0  0/4/0/0   td   CLAIMED     INTERFACE    HP Tachyon XL2 Fibre Channel Mass graphics/ccc.gif Storage Adapter                       /dev/td0 

The Instance number of the Fibre Channel card plays no part in the device file name for a disk; remember that a SCSI disk is attached to an ext_bus interface.

We can now talk about the address of the actual disks themselves. HP-UX hardware paths follow the SCSI-2 standard for addressing, so we use the idea of a target and SCSI logical unit number (LUN). The LUN number we see in Figure 4-1 is the LUN number assigned by the disk array. This is a common concept for disk arrays. HP-UX has to translate the LUN address assigned by the disk array into a SCSI address. If we look at the components of a SCSI address, we can start to try to work out how to convert a LUN number into a SCSI address:

  • Target : 4-bit address means valid values = 0 through 15

  • LUN : 3-bit address means valid values = 0 through 7

If we take an example of a LUN number of 20 10, there's a simple(ish) formula for calculating the SCSI target and LUN address. Here's the formula:

  1. Ensure that the LUN number is represented in decimal; an HP XP disk array uses hexadecimal LUN numbers while an HP VA disk array uses decimal LUN numbers.

  2. Divide the LUN number by 8. This gives us the SCSI target address.

  3. The remainder gives us the SCSI LUN.

    For our simple example of a LUN=20:

  4. LUN(on disk array) = 20 10.

  5. Divide 20 by 8 SCSI target = 2.

  6. Remainder = 4 SCSI LUN = 4.

  7. SCSI address = 2.4.

This would give us a complete hardware path of:

 

  0/2/0/0  .  1  .  1  .   .   .  2  .  4  

Component of address

HBA/Interface card

Domain (=switch number)

Area (=physical port number)

Port (always 0 in a switched fabric)

Virtual SCSI Bus

SCSI target

SCSI LUN


The problem we've got here is that LUN numbers assigned by disk arrays can exceed 128! This poses an interesting problem for HP-UX. The way we get around that is simply to increase the Virtual SCSI Bus address every time we go beyond a multiple of 128. This goes to explain the two ext_bus we can see in the ioscan output above:

 

  ext_bus  15  0/2/0/0.2.1.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  20  0/2/0/0.2.2.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  19  0/4/0/0.2.1.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface   ext_bus  23  0/4/0/0.2.2.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface  

The original ext_bus interfaces ran out of address space because we have a LUN number of 200 on our disk array. HP-UX will create additional ext_bus interfaces by increasing the Virtual SCSI Bus address by 1. These new interfaces are assigned their own Instance numbers. The Virtual SCSI Bus address is a 4-bit value, so we can support 16 VSB interfaces x 128 LUNS per VSB = 2048 LUNs per physical Fibre Channel card. This has a bearing on our formula for calculating our SCSI target and SCSI LUN address. We now have to accommodate the Virtual SCSI Bus in our formula. Here goes:

  1. Ensure that the LUN number is represented in decimal; an HP XP disk array uses hexadecimal LUN numbers while an HP VA disk array uses decimal LUN numbers.

  2. Virtual SCSI Bus starts at address 0.

  3. If LUN number is < 128 10, go to step 7, below.

  4. Increment Virtual SCSI Bus by 1 10 .

  5. Subtract 128 10 from the LUN number.

  6. If LUN number is still greater than 128 10, go back to step 4 above.

  7. Divide the LUN number by 8. This gives us the SCSI target address.

  8. The remainder gives us the SCSI LUN.

In our SAN, we have a LUN = 200. Let's work out the three components of the SCSI address:

  1. LUN(on disk array) = 200 10

  2. Virtual SCSI Bus = 0

  3. Is LUN number > 128 10 ?

    1. YES

  4. Virtual SCSI Bus = 1

  5. LUN number = 200 10 “ 128 10 = 72 10

  6. Is LUN number > 128 10 ?

    1. NO

  7. Divide 72 by 8 SCSI target = 9

  8. Remainder = 0 SCSI LUN = 0

  9. SCSI address = 1.9.0

We can now work out how specific device files are created. Let's take an example disk array LUN=200 connected via the Fibre Channel card at H/W Path 0/2/0/0. We know the ext_bus interface can see the LUN through two ports on Switch2 (ports 1 and 2). Because we are dealing with LUN=200 which exceeds 128, we are dealing with the ext_bus interfaces which utilize a new Virtual SCSI Bus interface:

 

 ext_bus  15  0/2/0/0.2.1.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface ext_bus  20  0/2/0/0.2.2.0.1     fcparray CLAIMED     INTERFACE    FCP Array Interface 

As you can see, the Virtual SCSI Bus address has been increased by one, giving us a new interface and consequently additional Instance numbers. The Instance numbers for these two new interfaces are 15 and 20 respectively, giving device file components, c15 and c20 . From our formula, to calculate the SCSI target and SCSI LUN addresses, we now know the other two components of the device file name, t9 and d0 . Here's an extract from an ioscan that shows these disks and their associated device file names :

 

 root@hp1[]  ioscan -fnC disk  Class     I  H/W Path        Driver   S/W State   H/W Type     Description =========================================================================== ... disk     15  0/2/0/0.2.1.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c15t9d0   /dev/rdsk/c15t9d0 ... disk     19  0/2/0/0.2.2.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c20t9d0   /dev/rdsk/c20t9d0 

I must admit that this was not entirely straightforward when I first looked at it, especially when you have a server connected to a large disk array with hundreds of LUNs, each with multiple paths to it. In the example above, we have a single LUN (=200) configured on a disk array, which has four paths to it. Trying to identify it can be a long and complex task. I would remind you of the adb command we have seen earlier which read the LVM header off the beginning of the disk. This can prove very useful in identifying disks that have multiple PV links to them.

IMPORTANT

Understanding the importance of the N_Port ID and Virtual SCSI Bus is crucial to deciphering HP-UX hardware paths for any SAN attached devices.


Below is the ioscan output for the node hp1 with all the paths and device files for the five LUNs in Figure 4-1:

 

 root@hp1[]  ioscan -fnC disk  Class     I  H/W Path        Driver   S/W State   H/W Type     Description =========================================================================== disk      0  0/0/1/1.15.0    sdisk    CLAIMED     DEVICE       HP 36.4GST336753LC                             /dev/dsk/c1t15d0   /dev/rdsk/c1t15d0 disk      1  0/0/2/1.15.0    sdisk    CLAIMED     DEVICE       HP 36.4GST336753LC                             /dev/dsk/c3t15d0   /dev/rdsk/c3t15d0 disk     20  0/2/0/0.1.1.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6188A                             /dev/dsk/c10t0d0   /dev/rdsk/c10t0d0 disk     21  0/2/0/0.1.1.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6188A                             /dev/dsk/c10t0d1   /dev/rdsk/c10t0d1 disk      9  0/2/0/0.2.1.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c14t0d0   /dev/rdsk/c14t0d0 disk     10  0/2/0/0.2.1.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c14t0d1   /dev/rdsk/c14t0d1 disk     15  0/2/0/0.2.1.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c15t9d0   /dev/rdsk/c15t9d0 disk     16  0/2/0/0.2.2.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c18t0d0   /dev/rdsk/c18t0d0 disk     17  0/2/0/0.2.2.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c18t0d1   /dev/rdsk/c18t0d1 disk     19  0/2/0/0.2.2.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c20t9d0   /dev/rdsk/c20t9d0 disk     11  0/4/0/0.1.1.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6188A                             /dev/dsk/c11t0d0   /dev/rdsk/c11t0d0 disk     12  0/4/0/0.1.1.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6188A                             /dev/dsk/c11t0d1   /dev/rdsk/c11t0d1 disk     13  0/4/0/0.2.1.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c16t0d0   /dev/rdsk/c16t0d0 disk     14  0/4/0/0.2.1.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c16t0d1   /dev/rdsk/c16t0d1 disk     18  0/4/0/0.2.1.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c19t9d0   /dev/rdsk/c19t9d0 disk      2  0/4/0/0.2.2.0.0.0.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c4t0d0   /dev/rdsk/c4t0d0 disk      3  0/4/0/0.2.2.0.0.0.1   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c4t0d1   /dev/rdsk/c4t0d1 disk     22  0/4/0/0.2.2.0.1.9.0   sdisk    CLAIMED     DEVICE       HP      A6189A                             /dev/dsk/c23t9d0   /dev/rdsk/c23t9d0 root@hp1[] 

This should give you some examples to try to work out which disk Interface number is associated with each LUN, e.g., disk Interface = 22 (H/W Path = 0/4/0/0.2.2.0.1.9.0) is LUN 200 connected via /dev/td0 through the SAN via Switch2, port 2.



HP-UX CSE(c) Official Study Guide and Desk Reference
HP-UX CSE(c) Official Study Guide and Desk Reference
ISBN: N/A
EAN: N/A
Year: 2006
Pages: 434

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net