In-Band Virtualization

"In- Band " Virtualization

Host-based virtualization, as noted above, is only one of at least four discernable variants in the SAN virtualization game. A second group of approaches are referred to collectively as "in-band" or "in-the-wire" virtualization by the trade press and analyst community. This description, while it has the advantage of convenience by capturing several product implementation philosophies under single moniker, also confuses the issue. All SAN virtualization techniques are, by their nature, located in the data path (see Figure 7-5).

Figure 7-5. All virtualization is in-band.

In the case of virtualization software located on a host, the software layer is near the beginning of the I/O request, but still in the path. So-called in-band techniques are also in the path because they impact data traffic as it travels between the initiator of the I/O request and the target of that request, the storage platform. Current favorites for this type of virtualization are specialized "appliances" ”single-purpose servers running virtualization software and installed in the wire ”and "fat" switches ”storage switches enhanced with new software functionality for performing virtualization.

The third approach, ostensibly called "out-of-band" virtualization, is effectively classified as "in-band" as well. A specialized server appliance sits outside the data path, where it monitors the condition of volumes and determines when capacity must be added. When necessary, it initiates a device driver update process and rewrites the parameters of the device driver on each host in order to change the size of the virtual volumes that applications are able to use. The device driver acted upon by the appliance is in-band.

The final approach, virtualization on the controller of an array, is also an in-band approach, since the access made to backend storage is through the "front door" of the controller's virtualization engine. From the perspective of the data path, the distinction between in and out of band is meaningless.

Having said that, what we are calling in-band virtualization ”in the data path appliances and enhanced switch functionality ”is currently the hot spot for virtualization engineering. In 2003, it seemed that nearly every storage hardware manufacturer, and many storage management software vendors, were either establishing alliances with in-band appliance vendors or teaming with leading Fibre Channel and IP switch vendors to migrate virtualization software to those platforms.

Conceptually, it makes more sense to reduce the complexity of virtualization by vesting the functionality in a single, more easily managed and administered location such as a switch or appliance, rather than on hosts or storage arrays individually. A number of vendors seized upon this model early on, using commodity server hardware and operating system extensions to build appliance products.

Concerns were immediately raised about the potential saturation of the commodity server bus by placing it in the line between an expanding number of initiators and an expanding pool of targets. Said one IT manager for a large insurance company, "Queuing theory alone suggests that you can't push traffic from 800-plus servers communicating with 180 TB of disk organized into storage array platforms through what is essentially a PC bus without introducing a significant chokepoint and significant latency."

One out-of-band storage appliance vendor used a hypothetical math problem to demonstrate the difficulties "inherent" in its competitors ' in-band approaches:

As an example, in a SAN with 50 application servers, the throughput required for each can reach 100 MB/sec. This value, 100 MB/sec * 50 servers = 5 GB/sec. While it is fairly easy to install multiple RAID subsystems with aggregated performance exceeding this requirement, a typical symmetric appliance based on a standard Pentium III processor with a 64/66 PCI bus can deliver no more than 520 MB/sec (that is 260 MB/sec sustained rate as data needs to enter and exit through the PCI bus on the appliance) ”a far cry from the specified requirement. Needless to say, as the SAN becomes larger, this problem becomes more acute.

The hardware cost to construct [an in-band] appliance that will support 20 servers with reasonable performance could reach $25,000 and more; this cost is at least doubled if a High Availability configuration is required.

Some [in-band] designs are attempting to circumvent the performance problem through adding a cache facility to the appliance. This approach further loads the appliance's memory subsystem. For many reasons cache is much more effective when it is distributed (i.e., part of the RAID subsystem) rather than centralized (i.e., part of the storage management appliance).

The use of caching also complicates High Availability (HA) configurations, which try to avoid single points of failure, as well as Scalable configurations, where additional appliances are added to respond to increased load. If caching is used with multiple appliances, some kind of cache coherency strategy is required. Experience with RAID controllers shows that this can be very complicated (and expensive).

In order to achieve high performance on the server side, while using two (or more) HBAs, a software driver should be installed on each server. That feature is common to the [in-band and out-of-band] solutions. Both need software to be installed on the server in order to create redundant and high performance (multiple paths) solutions for increasing throughput. ^[1]

Thus, consumer speculation, reinforced by vendor infighting, has created an impression of in-band virtualization inefficacy that has, in turn , proven the most difficult hurdle for vendors of in-band appliances to surmount. In fact, virtualization performance is determined by three factors: processor efficiency (how quickly data can be processed ), parallelism (how many processes can be accomplished in parallel or at the same time), and data pathing (how optimized is the path used to move data to where it needs to go, among and between internal components such as cache memory and CPU, and through busses and networks). In the mainframe world, where vendors have enjoyed nearly complete control over the design of both equipment and operating systems, this control has been used to optimize system architecture specifically for virtualization, elevating virtualization in the process to the status of an architectural mainstay. In the realm of open systems, such single-vendor hegemony does not exist, and implementations have been uneven in terms of performance optimization.

Products like DataCore Software Corporation's SANSymphony, however, are maturing rapidly . This virtualization engine, which is part of the offerings of other vendors including Hitachi and Fujitsu SOFTEK, is designed to run on multiprocessor Storage Domain Servers (SDS), which do not have the interrupt processing requirements of general-purpose servers. According to the company,

Multi-processing SDSs continuously poll I/O channels for new activity (read or write requests ) using one or more very fast CPUs. I/O processing takes place uninterrupted using local memory as very high-speed disk block caches. At least one CPU in each SDS is set aside to handle interrupt-driven background management tasks that would otherwise interfere with I/O processing. Through these techniques, SANsymphony software overcomes I/O performance limits encountered by heavily interrupted hosts.

A configuration that might cap at 15,000 I/Os as an interrupt-heavy host can deliver upwards of 200,000 I/Os as an SDS. ^[2]

DataCore claims that its caching technique actually reduces the disk I/O latency typically experienced by applications. The vendor attributes this to the fact that SDS caches are fast, yet lower cost and scalable across the network. It also cites its built-in data prefetching scheme and read request sorting as an accelerator, rather than a decelerator of I/O.

Caching also hides some of the write delays experienced by applications storing data on slower devices. The caches acknowledge the write after quickly depositing the data in memory. In the background, software opportunistically coalesces several of these cached blocks destined for the same area of a disk into one physical I/O. Fewer and larger transfers replace multiple smaller I/Os to reduce the overhead on the back-end device and its channels.

Some observers fear data loss or data corruption from write caching. Robust write caching techniques have been around for years and have been shipping in volume with every major disk array. As with all the successful write caching implementations, SANsymphony replicates the I/O in real-time to an independent location before acknowledging completion. The redundant data is maintained on an entirely separate SDS node. This ensures full data access to an alternate image if the primary cached contents are unreachable. ^[3]

In summary, DataCore makes salient observations about its platform intended to answer the many detractors of in-band virtualization in the analyst community, trade press and marketing departments of competitors.

• Storage Domain Servers are generally three to four generations ahead of the processors and busses assembled in array controllers.

• SANsymphony software effectively uses multiple independent PCI busses in each Storage Domain Server to match the bandwidth needs of the storage pool. For example, each SDS node cited earlier in the performance reports drove five separate PCI busses, three of which accounted for 1.6 GB/sec of storage networking bandwidth.

• Sheer horsepower and economics make it possible to front-end premium priced arrays with low-cost Storage Domain Servers at a fraction of the price.

• Storage Domain Servers field most of the repetitive read requests right out of their high-speed caches to effectively offload the arrays.

• At the time of this writing, just two modestly configured Storage Domain Servers can handle 2 to 10 times the peak transaction and throughput of the world's most powerful arrays.

• Increasing the I/O budget for [DataCore] storage pools is as simple as nonintrusively adding another low-cost Storage Domain Server. That means more cache, more CPUs, more ports and more bandwidth can be quickly added to handle unpredictable demands. Basically, in-band sensing, end-to-end I/O path management and in-band caching have been shown to improve the behavior of a dynamic storage network. You can't offer the same benefits from the sidelines using an out-of-band appliance. ^[4]