In the world of high-availability, we are always concerned with a Single Point Of Failure (SPOF). We need to consider our interface cards as one such SPOF. The way we alleviate this issue is to employ Alternate PV Links . This is simply an additional interface card and additional cable connected to the disk. The hardware setup of an Alternate PV Link is crucial as well. If your interface is a SCSI interface, you will need to ensure that the SCSI target address of the interface card is unique. Having two devices on the SCSI bus with the same target address will cause no end of trouble (lots of SCSI lbolt / reset messages in syslog for one thing!). The problem with Alternate PV Links is the ability to spot them. From the perspective of HP-UX, it simply sees another disk drive: another entry from ioscan . The skill is being able to identify an Alternate PV Link in order to configure them into a volume group . Take this ioscan output from my system, for example: root@hpeos003[] ioscan -fknC disk more Class I H/W Path Driver S/W State H/W Type Description ====================================================================== disk 0 0/0/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t0d0 /dev/rdsk/c0t0d0 disk 1 0/0/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t1d0 /dev/rdsk/c0t1d0 disk 2 0/0/1/0.2.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t2d0 /dev/rdsk/c0t2d0 disk 3 0/0/1/0.3.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t3d0 /dev/rdsk/c0t3d0 disk 4 0/0/1/0.4.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t4d0 /dev/rdsk/c0t4d0 disk 5 0/0/1/0.5.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t5d0 /dev/rdsk/c0t5d0 disk 6 0/0/1/0.6.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c0t6d0 /dev/rdsk/c0t6d0 disk 7 0/0/1/1.15.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC /dev/dsk/c1t15d0 /dev/rdsk/c1t15d0 disk 8 0/0/2/1.15.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC /dev/dsk/c3t15d0 /dev/rdsk/c3t15d0 disk 9 0/6/0/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t8d0 /dev/rdsk/c4t8d0 disk 10 0/6/0/0.9.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t9d0 /dev/rdsk/c4t9d0 disk 11 0/6/0/0.10.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t10d0 /dev/rdsk/c4t10d0 disk 12 0/6/0/0.11.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t11d0 /dev/rdsk/c4t11d0 disk 13 0/6/0/0.12.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t12d0 /dev/rdsk/c4t12d0 disk 14 0/6/0/0.13.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t13d0 /dev/rdsk/c4t13d0 disk 15 0/6/0/0.14.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c4t14d0 /dev/rdsk/c4t14d0 disk 16 0/6/0/1.0.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t0d0 /dev/rdsk/c5t0d0 disk 17 0/6/0/1.1.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t1d0 /dev/rdsk/c5t1d0 disk 18 0/6/0/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t2d0 /dev/rdsk/c5t2d0 disk 19 0/6/0/1.3.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t3d0 /dev/rdsk/c5t3d0 disk 20 0/6/0/1.4.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t4d0 /dev/rdsk/c5t4d0 disk 21 0/6/0/1.5.0 sdisk CLAIMED DEVICE HP 73.4GST373307LC /dev/dsk/c5t5d0 /dev/rdsk/c5t5d0 root@hpeos003[] It would appear that I have a host of disks connected to interface 0/6/0/1. The device addresses of 1, 2, 3, 4, 5, and 6 look suspiciously similar to the devices connected to interface 0/0/1/0. If you know your hardware configuration well and it is well documented, then it may be obvious to you that the disks c5t1d0 , and so on, are actually Alternate PV Links to disk c0t1d0 , and so on. You need to be sure of your configuration because it wouldn't be the first time I had a customer who ran a pvcreate “f command against a device file that turned out to be an Alternate PV Link. You should be suspicious the first time you run pvcreate against such a device because it will warn you that the device already belongs to a volume group: root@hpeos003[] pvcreate /dev/rdsk/c5t1d0 pvcreate: The physical volume already belongs to a volume group root@hpeos003[] This should immediately raise our suspicions because this device is not listed in /etc/lvmtab : root@hpeos003[] strings /etc/lvmtab /dev/vg00 /dev/dsk/c1t15d0 /dev/dsk/c3t15d0 /dev/vgora1 /dev/dsk/c0t1d0 /dev/dsk/c0t2d0 /dev/dsk/c0t3d0 /dev/dsk/c4t9d0 /dev/dsk/c4t10d0 root@hpeos003[] The disk in question could have been an old disk, and pvcreate is simply saying that it has found a valid LVM header on the disk. We need to be absolutely sure that this is a separate disk before using pvcreate “f . Before even contemplating using pvcreate “f , we can at least read the LVM header of the disk using this and the existing device file. If the LVM header is different, then the disks are separate disks. If the LVM header is exactly the same, then the new device file is simply an Alternate PV Link to the same disk. Reading the LVM header off of the disk requires a little knowledge of the LVM structures at the beginning of the disk. That's where you'll find the 8KB header used to point to the boot area. We then have the PVRA . The first element of the PVRA is a LVM Record. That identifies this disk as an LVM disk. The LVM Record is 8 bytes in size . That's 8KB + 8 bytes = 8200(2008 in hex) bytes. If we read from that location, we will find four interesting numbers in the PVRA ; the CPU-ID , the PV-ID , the CPU-ID (again), and the VG-ID . If we were to read this information from two distinct device files and see exactly the same information, we are looking at an Alternate PV Link: root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c0t1d0 2008: 894960601 1067429315 894960601 1067430042 root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c5t1d0 2008: 894960601 1067429315 894960601 1067430042 root@hpeos003[] As we can see, here the information from both device files is exactly the same. I can confirm this with the other disks in the volume group: root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c0t2d0 2008: 894960601 1067429321 894960601 1067430042 root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c5t2d0 2008: 894960601 1067429321 894960601 1067430042 root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c0t3d0 2008: 894960601 1067429631 894960601 1067430042 root@hpeos003[] echo "0x2008?4D" adb /dev/dsk/c5t3d0 2008: 894960601 1067429631 894960601 1067430042 root@hpeos003[] We can safely conclude that the device c5t[123]d0 are Alternate PV Links for the corresponding device c0t[123]d0 . We can add the Alternate PV Links into the volume group: root@hpeos003[] vgextend /dev/vgora1 /dev/dsk/c5t[123]d0 Volume group "/dev/vgora1" has been successfully extended. Volume Group configuration for /dev/vgora1 has been saved in /etc/lvmconf/vgora1.conf root@hpeos003[] root@hpeos003[] vgdisplay -v vgora1 more --- Volume groups --- VG Name /dev/vgora1 VG Write Access read/write VG Status available Max LV 255 Cur LV 1 Open LV 1 Max PV 16 Cur PV 6 Act PV 6 Max PE per PV 17501 VGDA 12 PE Size (Mbytes) 4 Total PE 87495 Alloc PE 500 Free PE 86995 Total PVG 2 Total Spare PVs 1 Total Spare PVs in use 0 --- Logical volumes --- LV Name /dev/vgora1/db LV Status available/syncd LV Status available/syncd LV Size (Mbytes) 1000 Current LE 250 Allocated PE 500 Used PV 2 --- Physical volumes --- PV Name /dev/dsk/c0t1d0 PV Name /dev/dsk/c5t1d0 Alternate Link PV Status available Total PE 17499 Free PE 17249 Autoswitch On PV Name /dev/dsk/c0t2d0 PV Name /dev/dsk/c5t2d0 Alternate Link PV Status available Total PE 17499 Free PE 17499 Autoswitch On PV Name /dev/dsk/c0t3d0 PV Name /dev/dsk/c5t3d0 Alternate Link PV Status available Total PE 17499 Free PE 17499 Autoswitch On PV Name /dev/dsk/c4t9d0 PV Status available Total PE 17499 Free PE 17249 Autoswitch On PV Name /dev/dsk/c4t10d0 PV Status available/standby spare Total PE 17499 Free PE 17499 Autoswitch On PV Name /dev/dsk/c4t11d0 PV Status available Total PE 17499 Free PE 17499 Autoswitch On --- Physical volume groups --- PVG Name PVG0 PV Name /dev/dsk/c0t1d0 PV Name /dev/dsk/c0t2d0 PV Name /dev/dsk/c0t3d0 PVG Name PVG1 PV Name /dev/dsk/c4t9d0 PV Name /dev/dsk/c4t10d0 PV Name /dev/dsk/c4t11d0 root@hpeos003[] If something should happen to the Primary Link, e.g., the interface card fails, we should see LVM automatically move over to the Alternate Link . root@hpeos003[] more /var/adm/syslog/syslog.log ... Oct 30 17:28:57 hpeos003 vmunix: LVM: Path (device 0x1f001000 ) to PV 1 in VG 1 Failed! Oct 30 17:28:57 hpeos003 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000042096080), from raw device 0x1f001000 (with priority: 0, and current flags: 0x40) to raw device 0x1f051000 (with priority: 1, and current flags: 0x0). Oct 30 17:28:57 hpeos003 vmunix: LVM: Path (device 0x1f002000 ) to PV 1 in VG 1 Failed! Oct 30 17:28:57 hpeos003 vmunix: LVM: Path (device 0x1f003000 ) to PV 2 in VG 1 Failed! Oct 30 17:28:57 hpeos003 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000420a1040), from raw device 0x1f003000 (with priority: 0, and current flags: 0x40) to raw device 0x1f053000 (with priority: 1, and current flags: 0x0). Oct 30 17:28:58 hpeos003 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000042096840), from raw device 0x1f002000 (with priority: 0, and current flags: 0xc0) to raw device 0x1f052000 (with priority: 1, and current flags: 0x0). Oct 30 17:29:02 hpeos003 vmunix: LVM: Recovered Path (device 0x1f051000 ) to PV 0 in VG 1. Oct 30 17:29:02 hpeos003 vmunix: LVM: Recovered Path (device 0x1f052000 ) to PV 1 in VG 1. Oct 30 17:29:02 hpeos003 vmunix: LVM: Recovered Path (device 0x1f053000 ) to PV 2 in VG 1. Oct 30 17:29:02 hpeos003 vmunix: LVM: Restored PV 0 to VG 1. Oct 30 17:29:02 hpeos003 vmunix: LVM: Restored PV 1 to VG 1. Oct 30 17:29:02 hpeos003 vmunix: LVM: Restored PV 2 to VG 1. ... root@hpeos003[] If the fault is intermittent, you may see several messages in syslog as LVM bounces between Primary and Alternate Links . In such a situation, I would place a Hardware Call with my local Response Center for an engineer to investigate any potential hardware problems with my interface cards/cables/terminators. I want you to remember one important thing regarding Alternate PV Links: they are there to provide redundancy against the failure of an interface. Some people would like the Alternate PV Link to be used to improve overall IO performance. (It's another path to our data, so we could spread the IO burden over all interfaces.) LVM does not support this configuration. Other software would have to be used to provide such a solution, e.g., Veritas Volume Manager with Dynamic Multi Pathing, HP AutoPath/VA, or AutoPath/XP, but these require the use of a VA or XP disk array. Alternate PV Links are used only in the event of an interface failure. Period. |