Mirroring

The most basic form of duplication redundancy is data mirroring, a concept that has been a staple of data center operations for decades. Mirroring is a storing-level function that is abstracted from higher-level functions, such as file systems and databases. Mirroring provides continuous storage I/O operations after a hardware, software, or communications failure keeps data from being accessed from a storage target.

The discussion of mirroring in this chapter does not include remote-copy technology, which is discussed in Chapter 10. While mirroring and remote copy are often discussed and described as being part of the same technology, they are actually quite a bit different. The two are used for different purposes and have significantly different costs.

Mirroring is primarily a real-time function that creates two I/Os that are processed in parallel. Both copies of data created by a mirror are available online to the system that uses the mirrored storage.

Remote copy, on the other hand, creates secondary, sequentially processed I/Os that are forwarded to remote storage by a storage subsystem. The remote copy of the data is not intended to be accessed online by the system that created the data. Other systems at the remote location access the remote copies.

The topics covered in this section are

• Mirroring operators, targets, and mirrored pairs

• Disk mirroring

• Location of mirroring operators

• The mirroring function

• Overlapped read operations for performance

• Assumptions about distance and mirroring

Mirroring Operators, Targets, and Mirrored Pairs

To simplify the discussion about whether mirroring is done in software or hardware controllers, the term mirroring operator is used in this book to describe the mirroring function that is performed. Similarly, to simplify the discussion of whether the mirroring is being done to devices, subsystems, LUNs, volumes, or partitions, this book refers to the mirrored storage as storage targets. With mirroring, two storage targets receive duplicate commands from a mirroring operator; these targets are referred to as a mirrored pair.

A mirroring operator and a mirrored pair are illustrated in Figure 8-4.

Figure 8-4. A Mirroring Operator and a Mirrored Pair of Storage Targets

NOTE

It's not really quite right to throw in the concept of volume or partition with device, subsystem, and LUN by calling them targets. A volume or partition is not a target in the SCSI sense of the word. However, where mirroring is concerned, these things do define the specific location where data gets stored. This is part of what makes talking about storage so much funthere are different ways to skin or de-fur the same cat.

Disk Mirroring

Mirroring in storage networks today extends the original concept of disk mirroring that has been used for decades to provide online data redundancy. The idea of disk mirroring is very simple: if something goes wrong with a disk drive, the system continues working with the other drive.

The developers of RAID included disk mirroring in their work, calling it RAID level 1, as the most basic form of a redundant disk array. Mirroring is not only useful for data protection, but it also has some performance advantages that are discussed later in this chapter.

Location of Mirroring Operators

Mirroring operators can be positioned at almost any point along the I/O path between an application and storage targets. Operators can run in host-based volume-management software, device driver software, host bus adapter hardware, network devices, and switches and storage subsystems.

Mirroring in Host System Software

Data mirroring is often accomplished as a fundamental feature of an operating system, volume management, or device driver software. Mirroring in host software has the advantage of having no single point of failure in the I/O path, including failures to HBAs. When failures occur in one path or target, the mirroring operator continues working through the other I/O path and its target. Of course, CPU and memory resources are needed in host systems to support the function. This usually is not a problem, except in systems that are constrained by CPU performance or have insufficient memory for their application workload.

Host software mirroring also works with mirrored pair targets that are accessed on the same I/O path. Obviously, a path failure where the I/O path segment is a single point of failure would block access to both targets in the mirrored pair. Figure 8-5 compares host software mirroring with and without a single point of failure in the I/O path.

Figure 8-5. Mirroring in Host Software with and Without a Single Point of Failure in the I/O Path

NOTE

Not all operating systems support mirroring as a software function, although it is relatively easy to provide it. Windows 2000 and Windows XP Professional come to mind. While these products are sold as higher-level desktop environments, neither has an option to install this most basic data-protection functionality. You have to buy the server version of the product to get software mirroring.

Apparently, this was done to limit Microsoft's exposure to licensing violations where users take a disk mirror and install it in unlicensed systems. To further protect itself from "theft by mirror," Microsoft implemented mirroring in its server products in a way that prevented licensing information from being copied from the primary disk to the secondary disk. The result is that if the "wrong" drive fails, it's necessary to reload the license keys from the installation CDs before rebooting the server the next time. Just what you needa disk mirror that makes you reinstall software.

Mirroring in Host Storage Controllers

Mirroring is sometimes provided in HBAs and integrated host system storage I/O controllers. While there is nothing necessarily wrong with mirroring in host hardware, it does establish the HBA or I/O controller as a single point of failure.

Multiported controllers allow the I/O path to be split so that both targets of a mirrored pair are connected to a different bus or network connection. Figure 8-6 shows mirroring in a dual-ported HBA, where both targets of a mirrored pair are connected to separate ports on the HBA.

Figure 8-6. Mirroring in a Dual-Ported HBA

Mirroring in a Network Device or Switch

A new development in the evolution of storage networks puts the mirroring operator in the I/O path in a network device or switch. The mirroring function could be provided by software or hardware running as a storage application inside networking equipment. Many products in the market do this; they are referred to generically as SAN virtualization products. Chapter 12, "Storage Virtualization: The Power in Volume Management Software and SAN Virtualization Systems," discusses these products in more detail.

Mirroring in a Storage Subsystem

One of the most common locations for mirroring is within a disk subsystem. The mirrored pair in a subsystem is exported for use by host systems as a LUN address. Of course, this LUN address could be made available through dual ports for high-availability connections to host systems. In this case, there would be both mirroring in the subsystem for redundant data protection and multipathing between the subsystem and the host, providing redundant connectivity. Multipathing is discussed in more detail in Chapter 11, "Connection Redundancy in Storage Networks and Dynamic Multipathing."

Figure 8-7 shows a configuration where a mirrored pair inside a disk subsystem is exported as a LUN address by two different target addresses to a host with dual connections to the storage network.

Figure 8-7. A Mirrored Pair in a Disk Subsystem Exported Over Two Different Target/LUN Addresses

Mirroring can be used in other types of storage subsystems besides disk subsystems; for instance, it is possible to mirror backup operations to dual tape drives inside a tape library. However, the highly variable condition of tape media makes it more difficult to create an environment with dependable operations that meet performance expectations. While mirroring disks is a fundamental feature of most disk subsystems, it is not widely featured in tape subsystems.

NOTE

Mirroring with host software to two different storage targets using two different I/O paths should not be confused with multipathing. With multipathing, two different I/O paths are established to access a single storage target in a disk subsystem. To put a fine point on it, you can mirror data over two different paths to two different LUs in a storage subsystem, or you can use multipathing to access one LU over one of the paths. The topic of multipathing is explored in more detail in Chapter 11.

Multiple Mirrors and Multilevel Data Redundancy

The mirroring function can be done many times, in sequence, along the I/O path. In other words, there can be mirroring operators in host software, HBAs, network switches, and disk subsystems all working with the same initial data. For example, two different disk subsystems could export a mirrored pair of internal disk drives as a LUN. An upstream mirroring operator, such as an HBA or volume management software, could then mirror these two LUNs and make them appear as a single target to the host system. So, instead of writing data to a single target, as the host software would assume, the data is actually being written to four different disk drives. Figure 8-8 demonstrates this scenario.

Figure 8-8. Two Layers of Mirrors, Resulting in Four Copies of Data

The Mirroring Function

Mirroring is a storing-level function; in other words, it works with blocks, block commands, such as SCSI command descriptor blocks (CDBs), and block storage address spaces. In essence, mirroring receives a storage command, copies it, sends it twice to two different storage targets, and then waits for acknowledgments from both targets.

I/O Termination and Reinitiation

Mirroring can be thought of as a real-time process where a single storage command arrives at the mirroring operator and is immediately forwarded to the mirrored pair targets. When the mirroring operator is running in the host system, the generation of the duplicate I/O commands can take place prior to being passed to the SCSI layer of the I/O stack. In other words, the mirroring operator does not have to interpret SCSI commands. Instead, it receives an I/O request from the file system through internal system interfaces, and then it formulates a pair of requests through the SCSI device driver's application program interface (API) layer. Figure 8-9 illustrates a single I/O request being exchanged twice between a host software mirroring operator and device driver software.

Figure 8-9. Mirroring in Host System Software, Creating Duplicate Requests for the Device Driver's API

The situation is slightly different for mirroring operators in the network or in storage subsystems. In these cases, the original I/O command is received as a SCSI CDB that must be interpreted. Notice that the mirroring operator in this scenario is actually initiating a new set of commands to the mirrored pair targets. In other words, the mirroring operator functions as a target to the upstream controller-initiator and then changes roles to act as the initiator for mirrored I/Os sent to downstream targets.

This dual role is commonly talked about as the mirroring operator terminating the original SCSI CDB and reinitiating new SCSI CDBs for the mirrored pair. Figure 8-10 illustrates the termination of upstream CDBs (commands) and the reinitiation of two downstream CDBs.

Figure 8-10. A Mirroring Operator Terminating an Upstream SCSI CDB and Reinitiating Two Downstream CDBs

NOTE

This issue of SCSI termination and reinitiation also applies to RAID and virtualization operators that are outside host systems in storage networks. In general, it's a good idea to know where all SCSI operations start and where they terminate.

When mirroring is understood this way, it's easier to see some of the challenges in mixing different kinds of disk drive technologies in the same mirrored pair, such as a SCSI drive and a SATA drive. In this case, the original command is not just duplicated, it is duplicated and converted to the other technology's protocol.

Host software might have an easier time working with different types of disk technology, because it does not have to issue two different types of commands and manage two different types of error conditions. However, it does have to issue two different requests and interactions with the system kernel and storage device drivers.

Identically Sized Storage Address Spaces

Mirroring is a basic form of storage virtualization that makes identically-sized storage targets appear as a single target to an upstream controller. It is important that both targets in a mirrored pair have the same storage address spaces (capacity). A file system or database system that writes to the mirrored pair is not aware of the mirroring and, therefore, creates only a single storage request when it reads or writes data. This includes a determination of the block address within the storage address space. In other words, the file system or database system has no way of differentiating between two different-sized storage address spaces in a mirror, and, therefore, it is essential that both targets in a mirrored pair be the same size. It simply cannot work any other way.

Overlapped Read Operations for Performance

Now that mirroring fundamentals have been discussed sufficiently, it's time to explain that mirroring works slightly differently for reads than for writes. In general, writes work as described earlier, but reads can be implemented differently to get better performance.

When writing, it is essential that both targets in a mirrored pair perform the same operation; otherwise, the entire purpose of mirroring is at risk. Therefore, writes are performed more or less simultaneously.

Reading data only depends on working with a single target in a mirrored pair. There is no point in reading from two targets if one of the read operations will only be discarded. For that reason, reads are often performed as overlapping operations where the work load is distributed between the two targets in the mirrored pair. Figure 8-11 shows a hypothetical division of I/O requests to different storage targets.

Figure 8-11. Overlapped Reads from Targets in a Mirrored Pair

The result is that the compound effects of rotational latency and seek time can be reduced considerably in this process. Adding another disk arm and spindle to the I/O queue can make an enormous difference in system performance.

Assumptions About Distance and Mirroring

Many times people want to use mirroring over longer distances for disaster protection and business continuity purposes. The basic idea is to have one storage target running locally and the other one running some distance away.

This can work, but there are some important things to keep in mind. First, remember that mirroring operators duplicate all storage commands, which can take an enormous amount of LAN, MAN, or WAN bandwidth; this can be expensive. Second, mirroring is a real-time process that requires low latency and minimal transmission delays. High-throughput applications might not be able to perform as expected using extended-distance mirrors. That said, it's not uncommon to see IT organizations use mirroring to achieve data redundancy for distances ranging up to 10 miles.

Remote-copy or file-level replication technologies designed for long-distance data redundancy provide much better options for data redundancy over longer distances. Chapter 10 investigates this topic in much more detail.

NOTE

The statement about mirroring operators duplicating all storage commands is not 100% correct. It is possible to implement mirroring so that only one target in a mirrored pair receives read requests. An example of a product that can do this is the Veritas Volume Manager. Changing the READ POLICY allows administrators to restrict reads to the local mirror target, which means the remote target would receive only writes and a few other commands, such as status requests. The end result would be a significant drop in bandwidth requirements for the remote link. Note that this would not reduce the latency of write operations over the remote mirror link, but the cost of that link could be considerably less.