22.4 Multi-Path Storage

Storage Area Networks (SANs) are being implemented in many of the clusters that we see being built. This is fabulous for performance and flexibility, but can be a source of headaches when trying to track down disk statistics. One particular such statistic is "cross-RAD I/O". This happens in GS-series AlphaServers where the disk I/O originates in one QBB^[2] (or RAD^[3]) but travels to another RAD to be written to disk. This is to be avoided if possible by spreading enough I/O controllers amongst the QBBs, but unfortunately may not be possible to eliminate completely due to architectural restrictions (i.e., not enough PCI bus slots). To see how much of this sort of I/O is happening, you can use hwmgr to check:

 # hwmgr –get attribute current | grep -E "dev_base_name|cross_rad|path_xfer"

The output will look like this:

    dev_base_name = kevm    dev_base_name = dsk0    path_xfer_0 = 186    cross_rad_iocnt = 19    cross_rad_initiated_iocnt = 15    dev_base_name = dsk1    path_xfer_1 = 43    path_xfer_1 = 0    path_xfer_1 = 0    path_xfer_1 = 0    cross_rad_iocnt = 0    cross_rad_initiated_iocnt = 0 ...    dev_base_name = dsk2    path_xfer_1 = 167    cross_rad_iocnt = 9    cross_rad_initiated_iocnt = 9 ...

    dev_base_name = dsk3    cross_rad_iocnt = 8    cross_rad_initiated_iocnt = 8    dev_base_name = dsk4    cross_rad_iocnt = 14    cross_rad_initiated_iocnt = 14    dev_base_name = dsk5    path_xfer_1 = 0    path_xfer_1 = 119    cross_rad_iocnt = 5    cross_rad_initiated_iocnt = 5    dev_base_name = dsk6    path_xfer_1 = 123    cross_rad_iocnt = 8    cross_rad_initiated_iocnt = 8    dev_base_name = dsk7    path_xfer_1 = 43    path_xfer_1 = 0    cross_rad_iocnt = 0    cross_rad_initiated_iocnt = 0    dev_base_name = cdrom0    path_xfer_2 = 2    cross_rad_iocnt = 0    cross_rad_initiated_iocnt = 0    dev_base_name = dsk8    path_xfer_4 = 159639    cross_rad_iocnt = 8093    cross_rad_initiated_iocnt = 5253    dev_base_name = dsk9    path_xfer_4 = 364    cross_rad_iocnt = 99    cross_rad_initiated_iocnt = 99    dev_base_name = dsk10    path_xfer_4 = 619    cross_rad_iocnt = 315    cross_rad_initiated_iocnt = 306    dev_base_name = dsk11    path_xfer_6 = 747    cross_rad_iocnt = 248    cross_rad_initiated_iocnt = 239    dev_base_name = cdrom1    path_xfer_7 = 2    cross_rad_iocnt = 0    cross_rad_initiated_iocnt = 0    dev_base_name = mc0    path_xfer_1 = 0    path_xfer_1 = 0    dev_base_name = tape0    path_xfer_1 = 0    dev_base_name = tape1    dev_base_name = scp0    path_xfer_1 = 0    path_xfer_1 = 0    path_xfer_1 = 0    path_xfer_1 = 0

From the highlighted output above you can see that dsk8 has the most cross-RAD I/O. You can avoid some cross-RAD I/O by configuring each QBB with an HBA to the shared storage, but if you see lop-sided cross-RAD I/O, you may need to specify that certain I/O intensive applications run on particular QBBs or RADs using the runon (1) command. Further I/O tuning of this sort is beyond the scope of this book.

^[2]Quad Building Block

^[3]Resource Affinity Domain