Kernel Module

After successful configuration is achieved, you can move on to making the device function under Linux. This is done at the driver module level. First determine whether the system can support the new hardware.

The simplest way to begin is by examining the system to determine whether another similar device is attached. See the following example of how to make this determination in a Linux environment.

Scenario 5-1: Determine Whether a Similar Device Exists

In our example, we begin by adding "another" PCI Emulex HBA LP9802 to our existing SUSE Linux Enterprise Server 9 (ia64) - Kernel 2.6.5-7.97-default machine. Again, our first step must be to confirm whether the kernel can control the newly added device.

First, determine whether a similar device exists. Begin by confirming the running OS distribution, as demonstrated next.

atlorca4:~ # cat /etc/SuSE-release SUSE LINUX Enterprise Server 9 (ia64) VERSION = 9 atlorca4:~ # cat /etc/lsb-release LSB_VERSION="1.3"

Note that these files exist on almost all distributions of Linux. In Red Hat, the filename would be /etc/redhat-release. You should always check the file to confirm which operating system version is in use.

Understanding the exact OS with regards to library versions is the next step. In the next example, we depict the PCI devices with different commands to illustrate different ways to determine which PCI adapters we have installed. In the following examples, we are looking for Emulex HBAs.

atlorca4:~ # lspci 0000:00:00.0 Communication controller: Hewlett-Packard Company Auxiliary Diva Serial Port (rev 01) 0000:00:00.1 Serial controller: Hewlett-Packard Company Diva Serial [GSP] Multiport UART (rev 03) 0000:00:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01) 0000:00:03.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01) 0000:30:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01) 0000:40:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02) 0000:41:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01) 0000:41:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01) 0000:41:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) 0000:50:01.0 Fibre Channel: Emulex Corporation LP9802 Fibre Channel Adapter (rev 01) 0000:60:01.0 PCI bridge: Intel Corp. 21154 PCI-to-PCI Bridge 0000:61:04.0 USB Controller: NEC Corporation USB (rev 41) 0000:61:04.1 USB Controller: NEC Corporation USB (rev 41) 0000:61:04.2 USB Controller: NEC Corporation USB 2.0 (rev 02) 0000:61:05.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE] 0000:70:01.0 Fibre Channel: Emulex Corporation LP9802 Fibre Channel Adapter (rev 01)

Scenario 5-2: Mapping with /proc/ioports

Now that we have confirmed that existing devices match what we are trying to add, the next logical step is checking the installed driver version. In addition, we need to confirm that a driver exists for the device that is on the PCI bus. Yes, logic dictates that if the device is present, so is the driver, but that is not always the case. The easiest way to match a hardware device to the driver is by using /proc/ioports on IA64 or by using an open source tool called lshw found at http://ezix.sourceforge.net/software/lshw.html for IA64- or IA32-based machines. Scenario 5-2 shows examples of how to make the correlation through /proc/ioports to a given device.

atlorca4:~ # cat /proc/ioports 00000000-00000fff : PCI Bus 0000:00   00000060-0000006f : i8042   00000500-000005ff : 0000:00:03.0     00000500-000005ff : sym53c8xx   00000600-000006ff : 0000:00:03.1     00000600-000006ff : sym53c8xx 00001000-00001fff : PCI Bus 0000:08 00002000-00003fff : PCI Bus 0000:10 00004000-00005fff : PCI Bus 0000:20 00006000-00007fff : PCI Bus 0000:30   00006000-000060ff : 0000:30:01.0     00006000-000060ff : sym53c8xx 00008000-00009fff : PCI Bus 0000:40   00008000-000080ff : 0000:41:01.0     00008000-000080ff : sym53c8xx   00008100-000081ff : 0000:41:01.1     00008100-000081ff : sym53c8xx 0000a000-0000bfff : PCI Bus 0000:50 ***>-Hardware address   0000a000-0000a0ff : 0000:50:01.0 ***>-Hardware address     0000a000-0000a0ff : lpfc *******>-LP9802 driver 0000c000-0000dfff : PCI Bus 0000:60   0000c000-0000c0ff : 0000:61:05.0 0000e000-0000ffff : PCI Bus 0000:70 ***>-Hardware address   0000e000-0000e0ff : 0000:70:01.0 ***>-Hardware address     0000e000-0000e0ff : lpfc *******>-LP9802 driver

Note that determining the HBA is difficult because of the lack of detailed information. However, looking at the same data through lshw yields valuable information about the HBA, as well as other features of the server. In the following example of lshw, we only present the computer description, CPU, and Emulex HBAs to save space.

atlorca4:~/lshw-B.02.03 # lshw atlorca4     description: Computer     product: server rx7620     vendor: hp     serial: USE4415CHJ     capabilities: smbios-2.3 dmi-2.3     configuration: uuid=8785E28C-17A4-D811-964E-E240B01E4A4B   *-core        description: Motherboard        physical id: 0      *-firmware           description: BIOS           vendor: HP           physical id: 0           version: 000.018           size: 1MB           capacity: 11MB      *-memory           description: System memory           physical id: 2           size: 2016MB      *-cpu:0           product: Itanium 2           vendor: Intel Corp.           physical id: 4           bus info: cpu@0           version: 5           size: 1300MHz      *-cpu:1           product: Itanium 2           vendor: Intel Corp.           physical id: 5           bus info: cpu@1           version: 5           size: 1300MHz           capabilities: emulated hyperthreading ...

We skip a great deal to focus on the Fibre HBAs.

... *-fiber:0           description: Fibre Channel           product: LP9802 Fibre Channel Adapter           vendor: Emulex Corporation           physical id: 8           bus info: pci@50:01.0           logical name: scsi5           version: 01           width: 64 bits           clock: 66MHz           capabilities: bus_master cap_list scsi-host           configuration: driver=lpfc           resources: iomemory:10040000-10040fff \           iomemory:10041000-100410ff irq:75 *-fiber:1           description: Fibre Channel           product: LP9802 Fibre Channel Adapter           vendor: Emulex Corporation           physical id: 1           bus info: pci@70:01.0           version: 01           width: 64 bits           clock: 66MHz           capabilities: bus_master cap_list           configuration: driver=lpfc           resources: iomemory:30040000-30040fff \           iomemory:30041000-300410ff irq:83

Though the previous lshw and /proc/ioports examples indicate that we have identical hardware and a driver capable of running such hardware, we have two outstanding issues. The first issue concerns the driver version, and the second concerns adding hardware when a driver or the existing hardware does not match the new hardware.

Scenario 5-3: Determine Driver Version

To address the first issue, we need to discuss methods of finding a driver for a given device. Many methods exist, but due to their speed, the following two are our favorites. The fastest way to determine the driver version is to use the cat command on the driver instance, found in /proc filesystem. In Scenario 5-2, we know that the driver is lpfc, and that it falls under the SCSI layer. In the /proc/scsi/ directory, we find that the lpfc directory contains multiple instances (HBAs), as indicated in this scenario.

atlorca4:/proc/scsi/lpfc # cat 5 Emulex LightPulse FC SCSI 2.10f HBA: Emulex LightPulse LP9802 (2 Gigabit) on PCI bus 50 device 08 irq 75 SerialNum: P6C640BTEPZPLU Firmware Version: 1.01A2 (H2D1.01A2) Hdw: 2003806d VendorId: 0xf98010df Portname: 10:00:00:00:c9:3d:3f:c5 Nodename: 20:00:00:00:c9:3d:3f:c5 Link Up - Ready:    PortID 0x652d13    Fabric    Current speed 2G lpfc0t00 DID 652b13 WWPN 50:06:0e:80:03:4e:46:11 WWNN \ \50:06:0e:80:03:4e:46:11

Note that depending on the applicable driver, manufacturer, or both, the information contained in these files can vary greatly. In the following illustration, we moved to a different host running Red Hat Linux and QLogic HBAs.

The first step to checking driver and HBA types is to determine the OS generation, as in the following example:

[root@cmtlinp3 qla2300]# cat /etc/lsb-release LSB_VERSION="1.3" [root@cmtlinp3 qla2300]# cat /etc/redhat-release Red Hat Enterprise Linux ES release 3 (Taroon Update 4) Linux cmtlinp3.atl.hp.com 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST \ 2004 i686 i686 i386 GNU/Linux

The second step is to check the driver for the QLogic cards. Note that lshw is used to find the QLogic cards. Then we go to the /proc/scsi/qla2300 directory.

[root@cmtlinp3 qla2300]# pwd /proc/scsi/qla2300 [root@cmtlinp3 qla2300]# ll total 0 -rw-r--r--    1 root     root            0 Apr 1 13:33 1 crw-------    1 root     root     254,   0 Apr 1 13:33 HbaApiNode [root@cmtlinp3 qla2300]# cat 0 QLogic PCI to Fibre Channel Host Adapter for QLA2340        :         Firmware version: 3.03.01, Driver version 7.01.01 Entry address = e0834060 HBA: QLA2312 , Serial# S95189 Request Queue = 0x1f8f0000, Response Queue = 0x1f8e0000 Request Queue count= 512, Response Queue count= 512 Total number of active commands = 0 Total number of interrupts = 977 Total number of IOCBs (used/max) = (0/600) Total number of queued commands = 0     Device queue depth = 0x20 Number of free request entries = 19 Number of mailbox timeouts = 0 Number of ISP aborts = 0 Number of loop resyncs = 0 Number of retries for empty slots = 0 Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0 Host adapter:loop state= <READY>, flags= 0x860a13 Dpc flags = 0x1000000 MBX flags = 0x0 SRB Free Count = 4096 Link down Timeout = 008 Port down retry = 030 Login retry count = 030 Commands retried with dropped frame(s) = 0 Configured characteristic impedence: 50 ohms Configured data rate: 1-2 Gb/sec auto-negotiate SCSI Device Information: scsi-qla0-adapter-node=200000e08b1c15eb; scsi-qla0-adapter-port=210000e08b1c15eb; scsi-qla0-target-0=50060b00001a0b12; SCSI LUN Information: (Id:Lun)  * - indicates lun is not registered with the OS. ( 0: 0): Total reqs 298, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 1): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 2): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 3): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 4): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 5): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 6): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c, ( 0: 7): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,

As you can see, a great deal of information can be provided, and the amount depends on the driver author. If the driver exists, as shown previously, adding the new hardware can be an easy step without taking the server offline. However, the phrase "adding hardware online" needs to be clearly defined.

As used throughout the remainder of this chapter (and as in the phrase, "adding hardware online"), "online" means not having to reboot the OS. Although this can be a great thing, note that it currently is not possible in most (if any) PCI-based machines with the Linux OS. However, adding storage to an existing PCI controller is fully allowed by almost all vendors.

To add PCI devices on the fly, first, the Processor Dependent Code (PDC) must initiate a scan to address any PCI slots that have been filled, which is done on initial power up. After the scan is complete, a wake-up command must be sent to the BIOS/PDC to reinitialize the scan procedure and power up a PCI slot. This step is followed by a rescan of the OS's PCI bus. To reverse this process, the OS must be able to suspend any driver instance (not the entire module, which would halt all instances of the driver), allowing a PCI slot to be powered down. The capability to power down a PCI slot exists in most enterprise servers such as HPUX, Solaris, and AIX among others, but this feature has not been perfected in the same hardware platforms running Linux. Hopefully, it is just a matter of time before the systems running Linux have this capability. However, in the meantime, a reboot is needed to add a PCI device, as long as the driver exists or is noted in the /etc/sysconfig/kernel under the variable INITRD_MODULES or /etc/modules.conf.

When installing a new PCI device, it does not matter whether the driver is installed before or after the device is inserted into the PCI slot. But again, at the current time in Linux, most PCI buses do not detect a newly added PCI device without a reboot.

In the next examples, we discuss adding LUNs through the PCI bus over the HBAs. In one example, we demonstrate adding LUNs online, whereas in another example, we discuss adding LUNs when there is no existing device driver.