Common Functions of Storage Subsystems

Storage subsystems vary considerably in their designs and the choice of technologies that are inplmented in them. Yet, most storage subsystems share common characteristics and functions. The management techniques for accessing these functions may differ from product to product, but the architectural underpinnings are consistent. For instance, storage address space management is a basic function that can be controlled and managed many different ways.

One of the most important functional areas in storage subystem is the way the subsystem accommodates connections to multiple host systems and multiple HBAs in those systems. The sections on multiplexing I/Os and LUN masking are intended to give readers insight into ways enterprise subsystems can support a large number of host systems.

Storage Address Space Management

Of all the functions storage subsystems provide, the most important is managing the block storage address spaces of the storage devices that are part of the subsystem.

The techniques of mirroring, RAID, and striping, previously discussed in this chapter, are all examples of managing a storage address space. While it is easiest to discuss these types of storage virtualization techniques using an entire disk drive as the smallest granular address element, in fact storage address space manipulations are actually performed on logical partitions of disk drives.

Disk drives in a subsystem are partitioned into logical entities that are combined with other logical entities on other disk drives to form mirrors, stripes, and RAID arrays. A disk drive can have one or more partitions. Figure 5-9 shows a collection of five disk drives in a subsystem. The disk drives have been partitioned and combined to form six different virtual devices for use by host systems.

Figure 5-9. Disk Drive Partitions Forming RAID Arrays and Mirrors

Differentiating Arrays and Subsystems

The term array is often used incorrectly to refer to a disk subsystem, as in "We just purchased a new array for our Oracle systems." Although the context makes this statement clear enough, the term array refers to the combination of multiple storage address spaces configured for use with striping and mirroring technology (in other words, as a RAID array). A single subsystem can have multiple RAID arrays of different sizes and levels running concurrently.

The word subsystem is preferred when discussing a storage product built by integrating controllers, power supplies, networking interfaces, cache memory, internal buses, and disk drives. With DAS technology, the difference between RAID and subsystems was not as significant as it is with storage networking. In a storage network, a RAID array can be constructed from elements on different subsystems. A simple example of this is an array of subsystems, like the one shown in Figure 5-10.

Figure 5-10. Array of Subsystems

Accessing Exported Storage with LUN Addressing

The storage address spaces that are made available to systems are sometimes referred to as exported storage. In other words, systems use the storage that is exported by subsystems in a storage network.

Exported storage is often referred to by the SCSI term logical unit number (LUN), as in the sentence "I have three LUNs larger than a terabyte." Technically, LUNs are used to access storage devices and resources and are responsible for interpreting and exchanging SCSI protocol information with host controllers. The term LUN is not intended to be synonymous with exported storage. For instance, LUNs are also used to access storage devices that do not contain storage address spaces, such as robotic controllers in tape subsystems.

NOTE

Misusing the term LUN is an excellent example of the sort of terminology sloppiness that makes storage networking more difficult than it needs to be. Storage professionals have been known to explain that LUN is another word for virtual disk. Well, they're not the same thing at all. There is a huge difference between the way storage resources are accessed and their characteristics and functions. Calling exported storage a LUN is akin to saying that a phone number is the same thing as a house.

The problem is that the technology has evolved from physical entities to logical constructs that have made some old terms obsolete, while the old functional relationships have been preserved. For example, the aggregation of storage address spaces with parity RAID arrays forms "virtual disks," because they are used by the host system the same as physical disks. Of course, the virtual disk RAID array is an aggregation of disk partitions with redundancy built in. It is considerably more complex than a disk drive and has important implications for database and system administrators.

Which brings us back to the term LUN. This term actually works pretty well to identify a specific storage resource exported by a particular subsystem, as in the phrase "Sunshine, LUN 14" or "Stormy, LUN 32." It is important to be able to clarify whether you are talking about "this LUN 12" or "that LUN 12" when there are multiple LUN 12s to choose from in your environment. Developing naming conventions that include how LUNs are assigned to exported storage is a very good idea.

Try to avoid using the term LUN generically, as in the sentence "Put five LUNs together in a blender and mash them up with some fried WAN links to make a big#$! remote LUN sandwich."

Multiplexing I/Os, Link Oversubscription, and Fan-In

At first thought, it's normal to suppose that a subsystem port would expose a single LUN for host systems to use, but this is not the case. In most cases there are several LUNs, representing several different storage address spaces being accessed through a single port.

The fact is that gigabit-speed SAN connections and subsystem ports have more bandwidth than most systems can utilize; in fact, they have more bandwidth than several PC-based systems typically need and can support. If subsystem ports were dedicated to specific host-LUN connections, they would be idle most of the time.

Therefore, it is common for switch-to-subsystem links to carry multiple host-LUN connections simultaneously. This is called oversubscribing the link or fan-in and is illustrated in Figure 5-11.

Figure 5-11. SAN Fan-In: Multiple Host-LUN Connections on an Oversubscribed Switch-to-Subsystem Link

The amount of fan-in for any subsystem port depends on the I/O capabilities of the subsystem controller and the characteristics of the applications using it. Applications with light I/O requirements, such as most Windows servers, might be able to oversubscribe 15 or more host-LUN sessions. Other applications with heavy I/O requirements might limit oversubscription to four to six sessions; or they might even be assigned as a dedicated link.

NOTE

There are no set rules of thumb for oversubscribing switch-to-subsystem links. Some of the variables besides the application include the networking technology used and the capabilities of the subsystem. When planning to oversubscribe a switch/subsystem link, however, you should watch out for a few surprises.

First, you should plan for backup traffic, which is probably much heavier than the day-to-day I/O traffic that the applications generate.

Second, consider how oversubscription works with traffic management plans, including multi-pathing, zoning, or virtual SANs/virtual LANs (VSANs/VLANs). Obviously, if you plan to segregate the I/O traffic in the SAN, make sure you don't also plan to multiplex primary and secondary paths over a single switch-subsystem link.

LUN Masking

Most subsystems have the ability to prevent unwanted or accidental access to exported storage through a technique called LUN masking. LUN masking offers a way to segregate traffic at the location where the data is stored, which is a pretty good idea, but the details of the implementation need to be understood to avoid unexpected mistakes.

LUN masking in a subsystem is similar to filtering by routers in a network, but there are some tricky details. Like network filtering, the subsystem is configured to discard transmissions from host systems that are "masked" from accessing them. There are no prescribed methods for LUN masking implementation. For example, a subsystem could keep either an include-list or an exclude-list of host world-wide-names that it would use to determine if I/O operations would be allowed to enter the subsystem.

The tricky part of LUN masking is rooted in the orthogonal nature of connecting and storing functions in SANs. The lower-level connecting protocol carries half the identifying information needed for filtering, and the application layer storing protocol carries the other half. More explicitly, the host address that is located in the source_ID field of the network transmission header is processed by the network port logic. Also, the LUN subaddress that is located in the header of the SCSI application command descriptor block (CDB) is processed by a part of the storage controller logic. The processing would not be an issue if storage subsystem ports were not oversubscribed, but they are, and host systems that access certain LUNs through a particular port should be masked from accessing other LUNs through the same port. Figure 5-12 illustrates how two different host systems accessing storage through the same subsystem port are masked from accessing each other's storage.

Figure 5-12. LUN Masking: Two Host Systems Accessing Data Through a Single Subsystem Port Are Masked from Accessing Each Other's LUNs

LUN masking is sometimes implemented in a way that spoofs host systems from discovering that masked LUNs even exist. This technique is rooted in the SCSI protocol discovery process, in which a host system discovers all available target addresses (and LUNs) by issuing a SCSI inquiry command to all addresses. The protocol is designed so that all SCSI targets acknowledge the inquiry, sending back ID information including product name, firmware release levels, and attached LUNs. LUN masking spoofs hosts by not acknowledging certain LUNs to certain hosts. Systems that do not discover LUNs do not use them.

This approach is a bit like trying to stop telemarketing phone calls by having an unlisted number and using caller ID to screen calls before answering. However, this approach does not prevent access from systems to storage for other commands that the systems could conceivably send. Any entity on the SAN with access to a subsystem port has the ability to send any SCSI commands to it.

NOTE

It's a good thing SCSI-based LUN masking exists, but the situation could be much better than it is. In fact, there might never be a good solution for access control using SCSI discovery mechanisms because they were never designed for shared storage environments.

This is one of the reasons I get excited about virtual networking technologies for SANs. Similar to the way VLAN or VSAN switches work, subsystems could assign virtual networking IDs to storage address spaces and discard incoming transmissions that lack proper credentials. A consistent mechanism for traffic segregation among switches and subsystems would be truly valuable. It's certainly not strong security, but it's much better than spoofing the discovery process and letting everything else fly through unchecked.

Integrating virtual networking in a storage subsystem would require new designs that incorporate switching technology. OK, so what? Isn't this why the whole thing is called storage networking, anyway?

The SMI Standard and Container Management

The Storage Networking Industry Association (SNIA) began in the late 1990s as an association of storage networking vendors working on several fronts to advance the technology and markets for storage networking. From its inception, one of the primary goals of SNIA was finding ways to manage heterogeneous storage environments. The group's work has resulted in the Storage Management Initiative (SMI), which was defined by the release of its first specification (SMI-S) in the spring of 2003.

SMI divides management tasks into three areas: storage resource management, container management, and information management. Storage address space management is considered part of container management in SMI. Besides subsystem controller technology, SMI's container management also includes volume management software, RAID technology, and storage virtualization.

SNIA's SMI is based on the object-oriented Common Information Model (CIM) of the Distributed Management Task Force (DMTF). SMI provides the underpinnings for consistent management methods through consistent programmatic interfaces that are exported by SMI-compliant entities in a storage network. One of the most important advantages of SMI is that it does not require third-party management agents to be loaded.

Of course, the software behind the interfaces has to be developed and tested across the entire industrya process that will take several years and much hard work. Nonetheless, SNIA's goal is to attain basic SMI interoperability throughout the industry by the end of 2005.

Redundant Access to Subsystem Storage

A single exported storage address space can be accessed through two different subsystem ports to achieve redundant paths. For example, a single address space can be accessed with the same LUN ID through both ports in a dual-port subsystem. In other words, port redundancy in a subsystem is achieved by configuring LUN access through multiple subsystem ports.

The assignment of LUNs to primary and secondary ports can be done any number of ways, depending on the amount of storage traffic generated by applications and host systems. Of course, it is important to verify that the primary and secondary ports being used are actually on two different HBAs/FAs to avoid a single point of failure in the connection path. The topic of connection redundancy in SANs is explored in Chapter 11.