Adding LUNs via PCI
| Many different methods of adding devices are discussed throughout this book. However, the most important method is one in which the operating system and application can stay online. In this section, we discuss ways to add LUNs to a PCI bus device on the fly (online) and offline. In the following discussion, we shed some light on limitations with the Linux kernel SCSI source code from older kernels. The SCSI source code responsible for scanning SCSI devices is located at /usr/src/linux/drivers/scsi/scsi_scan.c and contains the code and parameters for spawning the scan on a given bus. With older kernels such as 2.4.9-e.10, the definitions for the BLIST_ parameters were listed as follows: #define BLIST_NOLUN 0x001 /* Don't scan for LUNs */ #define BLIST_FORCELUN 0x002 /* Known to have LUNs, force scanning */ #define BLIST_BORKEN 0x004 /* Flag for broken handshaking */ #define BLIST_KEY 0x008 /* Needs to be unlocked by special command */ #define BLIST_SINGLELUN 0x010 /* LUNs should better not be used in parallel */ #define BLIST_NOTQ 0x020 /* Buggy Tagged Command Queuing */ #define BLIST_SPARSELUN 0x040 /* Non consecutive LUN numbering */ #define BLIST_MAX5LUN 0x080 /* Avoid LUNS >= 5 */ #define BLIST_ISDISK 0x100 /* Treat as (removable) disk */ #define BLIST_ISROM 0x200 /* Treat as (removable) CD-ROM */ Today, with the 2.6.X kernels, the following items have been added or changed: #define BLIST_NOLUN 0x001 /* Only scan LUN 0 */ #define BLIST_LARGELUN 0x200 /* LUNs past 7 on a SCSI-2 device */ #define BLIST_INQUIRY_36 0x400 /* override additional length field */ #define BLIST_INQUIRY_58 0x800 /* ... for broken inquiry responses */ #define BLIST_NOSTARTONADD 0x1000 /* do not do automatic start on add */ #define BLIST_MS_SKIP_PAGE_08 0x2000 /* do not send ms page 0x08 */ #define BLIST_MS_SKIP_PAGE_3F 0x4000 /* do not send ms page 0x3f */ #define BLIST_USE_10_BYTE_MS 0x8000 /* use 10 byte ms before 6 byte ms */ #define BLIST_MS_192_BYTES_FOR_3F 0x10000 /* 192 byte ms page 0x3f request */ #define BLIST_REPORTLUN2 0x20000 /* try REPORT_LUNS even for SCSI- 2 devs #define BLIST_NOREPORTLUN 0x40000 /* don't try REPORT_LUNS scan (SCSI-3 devs) */ Within the previous fields, the most important thing to do if you are having issues with scanning storage is to confirm that the dev_info fields have been defined with the appropriate functions. In the 2.4.X kernels, the dev_info was contained within the scsi_scan.c file; however, in the later kernels, the list of supported devices has become very lengthy and deserving of its own file, scsi_devinfo.c. Either way, the following example holds true for depicting device definitions according to the limits of scsi_scan.c with respect to kernel version. Throughout the remainder of this chapter, we use an HP XP1024 or XP128 storage array, in which devices are defined as open-X, where X is a value according to the defined array group. Within the scsi_scan.c or scsi_devinfo.c file, you should find a field containing the following: {"HP", "OPEN-", "*", BLIST_SPARSELUN}, /* HP XP Arrays */We now know the reason why we only find LUNs 07 (no greater than LUN 8). It is due to the missing BLIST_LARGELUN function. To correct the problem, correct the line as follows: {"HP", "OPEN-", "*", BLIST_SPARSELUN | BLIST_LARGELUN}, /* HP XP Arrays */If only it were that simple. . . . Well, yes, it is that simple if the kernel source or SCSI source in the working kernel source is up to date. Unfortunately, the older Red Hat AS 2.1 server, defined in this chapter as Machine 1, is running an outdated SCSI source tree. Remember, each time that a change is made in the source, you must recompile and boot the new kernel. Now that we have discussed some brief limitations of the scsi_scan source with respect to the device information section, let us discuss another scenario. In the following example, we refer to Machine 1, which is attached to the storage array previously mentioned. The array has only two LUNs presented through one storage port into a fabric. The host has two HBAs logging into the fabric, so it has twice the opportunity to see the devices. For simplicity, in the following example, assume that the zoning and LUN assignments are correct on the array. In the example, we have added two LUNs with hex values 7 and 8 to the storage and proceeded to remove and add the Emulex driver to call the scsi_scan code manually. Note that this is an offline procedure because all storage, if any existed, would go offline after the driver is removed. [root@cyclops scsi]# dmesg The kernel reports no information. [root@cyclops scsi]# insmod lpfcdd Using /lib/modules/2.4.9-e.10custom-gt/kernel/drivers/scsi/lpfcdd.o Warning: loading /lib/modules/2.4.9-e.10custom-gt/kernel/drivers/ scsi/lpfcdd.o will taint the kernel: no license You should ignore the taint message because it is just a notification that the driver or module does not adhere to the open source guidelines. [root@cyclops scsi]# dmesg Emulex LightPulse FC SCSI/IP 4.20p PCI: Enabling device 01:03.0 (0156 -> 0157) !lpfc0:045:Vital Product Data Data: 82 23 0 36 !lpfc0:031:Link Up Event received Data: 1 1 0 0 PCI: Enabling device 01:04.0 (0156 -> 0157) !lpfc1:031:Link Up Event received Data: 1 1 0 0 scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17 scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27 As depicted previously, no LUNs were found. Next, we could proceed to determine whether this is a fabric zoning issue or an array LUN assignment problem. Note that in the previous test, however, we are trying to depict a true code limitation. With Machine 1, we can depict the limitation in the scsi_scan.c source code by reviewing the definitions set forth in the code. An example of this limitation follows: #define BLIST_NOLUN 0x001 /* Don't scan for LUNs */ #define BLIST_FORCELUN 0x002 /* Known to have LUNs, force scanning */ #define BLIST_BORKEN 0x004 /* Flag for broken handshaking */ #define BLIST_KEY 0x008 /* Needs to be unlocked by special command */ #define BLIST_SINGLELUN 0x010 /* LUNs should better not be used in parallel */ #define BLIST_NOTQ 0x020 /* Buggy Tagged Command Queuing */ #define BLIST_SPARSELUN 0x040 /* Non consecutive LUN numbering */ #define BLIST_MAX5LUN 0x080 /* Avoid LUNS >= 5 */ #define BLIST_ISDISK 0x100 /* Treat as (removable) disk */ #define BLIST_ISROM 0x200 /* Treat as (removable) CD-ROM */ Note that BLIST_LARGELUN does not exist because it was added some time around the 2.4-21 kernel. BLIST_LARGELUN allows for device scanning over seven LUNs. However, the BLIST_SPARSELUN exists but did not help us find our sparse LUNs because LUN 0 must exist to start scanning the bus. Some drivers build a pseudo LUN 0 so that this condition never exists; however, most system administrators argue that skipping LUN 0 is not helpful. Nevertheless, LUN 0 does not have to exist if different scanning software is implemented. By using scsidev-2.35 scsidev -l, we can bypass this limitation and find our disk on the fly. At this point in the process, keeping track of the devices becomes substantially more difficult. In the following example, only Hex LUN value 8 exists on the array. Because LUN 0 is not defined, the BLIST_SPARSELUN is not called, so dmesg reports no new disk. After you issue the insmod lpfcdd commands, the following results are found in dmesg. Emulex LightPulse FC SCSI/IP 4.20p PCI: Enabling device 01:03.0 (0156 -> 0157) !lpfc0:045:Vital Product Data Data: 82 23 0 36 !lpfc0:031:Link Up Event received Data: 1 1 0 0 PCI: Enabling device 01:04.0 (0156 -> 0157) !lpfc1:031:Link Up Event received Data: 1 1 0 0 scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17 scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27 Again, no LUNs were found. Even by running the following command, we cannot get around this issue because it uses the same scsi_scan.c code. [root@cyclops scsi]# /tmp/scsidev-2.35/scsidev /proc/scsi/scsi extensions not found. Fall back to scanning. After the full scan starts, scsidev creates a symbolic device pointer that points to the internal SCSI disk /dev/sda. Note, however, that we do not need or use this device link in our servers. We have truncated a lot just to show the external SCSI disk. Found /dev/scsi/sgh0-2800c0i0l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i1l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i2l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i3l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i4l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i5l0 (Type 00) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i11l0 (Type 03) on sym53c8xx-1.7.3c-20010512 Found /dev/scsi/sgh0-2800c0i15l0 (Type 03) on sym53c8xx-1.7.3c-20010512 However, note that by adding a LUN 0 to the array, we can get around this problem, as indicated next. [root@cyclops scsi]# rmmod lpfcdd [root@cyclops scsi]# insmod lpfcdd [root@cyclops scsi]# dmesg ... Emulex LightPulse FC SCSI/IP 4.20p PCI: Enabling device 01:03.0 (0156 -> 0157) !lpfc0:045:Vital Product Data Data: 82 23 0 36 !lpfc0:031:Link Up Event received Data: 1 1 0 0 PCI: Enabling device 01:04.0 (0156 -> 0157) !lpfc1:031:Link Up Event received Data: 1 1 0 0 scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17 scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Vendor: HP Model: OPEN-9-CVS Rev: 2111 Type: Direct-Access ANSI SCSI revision: 02 Attached scsi disk sdg at scsi2, channel 0, id 0, lun 0 Attached scsi disk sdh at scsi2, channel 0, id 0, lun 1 Attached scsi disk sdi at scsi2, channel 0, id 0, lun 10 Attached scsi disk sdj at scsi3, channel 0, id 0, lun 0 Attached scsi disk sdk at scsi3, channel 0, id 0, lun 1 Attached scsi disk sdl at scsi3, channel 0, id 0, lun 10 SCSI device sdg: 103680 512-byte hdwr sectors (53 MB) sdg: unknown partition table SCSI device sdh: 103680 512-byte hdwr sectors (53 MB) sdh: unknown partition table SCSI device sdi: 103680 512-byte hdwr sectors (53 MB) sdi: unknown partition table SCSI device sdj: 103680 512-byte hdwr sectors (53 MB) sdj: unknown partition table SCSI device sdk: 103680 512-byte hdwr sectors (53 MB) sdk: unknown partition table SCSI device sdl: 103680 512-byte hdwr sectors (53 MB) sdl: unknown partition table Having a LUN 0 or sparse LUNs is dependent on the driver and the parameters set forth in the scsi_scan.c file. Though each setting has a large impact, no impact is as great as when a single LUN is added to the PCI bus, which causes applications to crash. The next example demonstrates how a single LUN can have such an enormous impact. We first must cover some basics. Again in this example, we use a few LUNs from a storage array. The LUN values are 0, 1, 2, 3, 10, and 11. There is no particular reason for the LUN values to be scattered as such, but any system administrator will run into this configuration at some point. With this setup, the user needs to be made aware of several issues that must be handled appropriately. For example, one issue is to determine how many targets an HBA sees. [root@cyclops tmp]# cat /proc/scsi/lpfc/2 Emulex LightPulse LPFC Driver Version: 2.01g HBA: Emulex LightPulse LP8000 1 Gigabit PCI Fibre Channel Adapter SerialNum: 2R04714316 Firmware Version: 3.82A1 (D2D3.82A1) Hdw: 2002506d VendorId: 0xf80010df Portname: 10:00:00:00:c9:24:13:27 Nodename: 20:00:00:00:c9:24:13:27 Link Up - Ready: PortID 0x7b0f00 Fabric Current speed 1G lpfc0t00 DID 770d00 WWPN 50:00:60:e8:02:ea:81:00 WWNN 50:00:60:e8:02:ea:81:00 <---Target is an XP48 port CL1-A lpfc0t01 DID 780500 WWPN 50:06:0e:80:03:4e:46:05 WWNN 50:06:0e:80:03:4e:46:05 <---Target is an XP128 port CL1-F Do not forget to look at all targets presented by every HBA, as continued next: [root@cyclops tmp]# cat /proc/scsi/lpfc/3 Emulex LightPulse LPFC Driver Version: 2.01g HBA: Emulex LightPulse LP8000 1 Gigabit PCI Fibre Channel Adapter SerialNum: 0000c9286bda Firmware Version: 3.90A7 (D2D3.90A7) Hdw: 2002506d VendorId: 0xf80010df Portname: 10:00:00:00:c9:28:6b:da Nodename: 20:00:00:00:c9:28:6b:da Link Up - Ready: PortID 0x7f0d00 Fabric Current speed 1G lpfc1t00 DID 770d00 WWPN 50:00:60:e8:02:ea:81:00 WWNN 50:00:60:e8:02:ea:81:00 <---Target is an XP48 port CL1-A lpfc1t01 DID 780500 WWPN 50:06:0e:80:03:4e:46:05 WWNN 50:06:0e:80:03:4e:46:05 <---Target is an XP128 port CL1-F Though this is not a true dual-path configuration (because it has only a single storage path), it still is a good illustration of the following fault. As shown previously, the HBA in question must be connected to a fabric because of 24-bit FCID 0x7b0f00. However, the dmesg and /var/log/ messages only show LUNs from a single target. Attached scsi disk sdg at scsi2, channel 0, id 1, lun 0 Attached scsi disk sdh at scsi2, channel 0, id 1, lun 1 Attached scsi disk sdi at scsi2, channel 0, id 1, lun 2 Attached scsi disk sdj at scsi2, channel 0, id 1, lun 3 Attached scsi disk sdk at scsi2, channel 0, id 1, lun 10 Attached scsi disk sdl at scsi2, channel 0, id 1, lun 11 Attached scsi disk sdm at scsi3, channel 0, id 1, lun 0 Attached scsi disk sdn at scsi3, channel 0, id 1, lun 1 Attached scsi disk sdo at scsi3, channel 0, id 1, lun 2 Attached scsi disk sdp at scsi3, channel 0, id 1, lun 3 Attached scsi disk sdq at scsi3, channel 0, id 1, lun 10 Attached scsi disk sdr at scsi3, channel 0, id 1, lun 11 All the previous LUNs are on target 1 rather than target 0. After adding a few LUNs on target 0, we simply need to run the following command: echo "scsi add-single-device 2 0 00 01" >/proc/scsi/scsi Upon reviewing dmesg, we find that the device was added and bound to /dev/sds. scsi singledevice 2 0 0 1 blk: queue dfb84a1c, I/O limit 4095Mb (mask 0xffffffff) Vendor: HP Model: OPEN-E-CM Rev: 0119 Type: Direct-Access ANSI SCSI revision: 03 blk: queue dfb8461c, I/O limit 4095Mb (mask 0xffffffff) Attached scsi disk sds at scsi2, channel 0, id 0, lun 1 SCSI device sds: 28452960 512-byte hdwr sectors (14568 MB) sds: After careful review, we add two other disks, LUNs 6 and 2, in that order: echo "scsi add-single-device 2 0 00 06" >/proc/scsi/scsi dmesg prints the following for LUN 6: scsi singledevice 2 0 0 6 blk: queue e5c65c1c, I/O limit 4095Mb (mask 0xffffffff) Vendor: HP Model: OPEN-3*2 Rev: 0119 Type: Direct-Access ANSI SCSI revision: 03 blk: queue e5c65e1c, I/O limit 4095Mb (mask 0xffffffff) Attached scsi disk sdt at scsi2, channel 0, id 0, lun 6 SCSI device sdt: 9613440 512-byte hdwr sectors (4922 MB) sdt: unknown partition table echo "scsi add-single-device 2 0 00 02" >/proc/scsi/scsi dmesg prints the following for LUN 2: scsi singledevice 2 0 0 2 blk: queue e5c6501c, I/O limit 4095Mb (mask 0xffffffff) Vendor: HP Model: OPEN-E Rev: 0119 Type: Direct-Access ANSI SCSI revision: 03 blk: queue e5c6521c, I/O limit 4095Mb (mask 0xffffffff) Attached scsi disk sdu at scsi2, channel 0, id 0, lun 2 SCSI device sdu: 28452960 512-byte hdwr sectors (14568 MB) sdu: unknown partition table The new list of devices is as follows: Attached scsi disk sdt at scsi3, channel 0, id 0, lun 1 Attached scsi disk sdg at scsi2, channel 0, id 1, lun 0 Attached scsi disk sdh at scsi2, channel 0, id 1, lun 1 Attached scsi disk sdi at scsi2, channel 0, id 1, lun 2 Attached scsi disk sdj at scsi2, channel 0, id 1, lun 3 Attached scsi disk sdk at scsi2, channel 0, id 1, lun 10 Attached scsi disk sdl at scsi2, channel 0, id 1, lun 11 Attached scsi disk sdm at scsi3, channel 0, id 1, lun 0 Attached scsi disk sdn at scsi3, channel 0, id 1, lun 1 Attached scsi disk sdo at scsi3, channel 0, id 1, lun 2 Attached scsi disk sdp at scsi3, channel 0, id 1, lun 3 Attached scsi disk sdq at scsi3, channel 0, id 1, lun 10 Attached scsi disk sdr at scsi3, channel 0, id 1, lun 11 Attached scsi disk sds at scsi2, channel 0, id 0, lun 1 Attached scsi disk sdt at scsi2, channel 0, id 0, lun 6 Attached scsi disk sdu at scsi2, channel 0, id 0, lun 2 The biggest problem is with LUNs 6 and 2 at the bottom of the list. The next time that the host is rebooted, the LUNs swap sdt and sdu values to the hard disk map point. This type of problem is why Linux developers created the scsidev project. After running scsidev, /dev/scsi block devices are built and listed by the SCSI interface, channel, ID, and LUN value as follows: brw-rw---- 1 root disk 65, 32 Apr 10 20:13 sdh2-0c0i0l1 brw-rw---- 1 root disk 65, 64 Apr 10 20:13 sdh2-0c0i0l2 brw-rw---- 1 root disk 65, 48 Apr 10 20:13 sdh2-0c0i0l6 brw-rw---- 1 root disk 8, 96 Apr 10 20:16 sdh2-0c0i1l0 brw-rw---- 1 root disk 8, 112 Apr 10 20:16 sdh2-0c0i1l1 brw-rw---- 1 root disk 8, 160 Apr 10 20:16 sdh2-0c0i1l10 brw-rw---- 1 root disk 8, 176 Apr 10 20:16 sdh2-0c0i1l11 brw-rw---- 1 root disk 8, 128 Apr 10 20:16 sdh2-0c0i1l2 brw-rw---- 1 root disk 8, 144 Apr 10 20:16 sdh2-0c0i1l3 brw-rw---- 1 root disk 8, 192 Apr 10 20:16 sdh3-1c0i1l0 brw-rw---- 1 root disk 8, 208 Apr 10 20:16 sdh3-1c0i1l1 brw-rw---- 1 root disk 65, 0 Apr 10 20:16 sdh3-1c0i1l10 brw-rw---- 1 root disk 65, 16 Apr 10 20:13 sdh3-1c0i1l11 brw-rw---- 1 root disk 8, 224 Apr 10 20:16 sdh3-1c0i1l2 brw-rw---- 1 root disk 8, 240 Apr 10 20:16 sdh3-1c0i1l3 Of course, modify the /etc/fstab accordingly so that if the /dev/sdX changes after a reboot, the /dev/scsi/sdhX-XcXiXlX does not. The new LUN has been found at /dev/scsi/sgh2-0c0i0l0, and a symbolic link has been made at /dev/sdq with the alternate path on /dev/sdj for block control. The nice feature of using scsidev to find new devices is that you don't have to worry about the LUN order. Think of it this way. A deck of cards has been stacked neatly on a table. If you remove the card on the bottom of the stack, the entire deck shifts down. In normal circumstances, this is of no concern. However, let's assume that a number has been written upon each card. The cards are numbered in order from 0 to 52, starting from the bottom. If you remove the number two card, does the number three card become number two? No, the value written on the card stays the same. That is the function that scsidev provides. Using the standard device addressing in Linux, the number three card becomes number two when the number two card is removed, which causes a great mess in /etc/fstab and any user application hardbound to a SCSI device file. |
EAN: 2147483647
Pages: 129