Remote Copy Application Processing

Remote copy applications use storing level functions that work with SCSI processes, commands, and data. This means that primary, secondary, and tertiary storage will have identical storage address spaces, because the commands forwarded by remote copy specify the block address where the update is to be made.

If the remote site storage needs to be used to access data after a disaster, a system will need to connect to the storage and mount it as local storage. This system will need to be able to access and use the same filing system as the one that is used by the application system at the primary site.

The Remote Copy System

It helps to think of the workings of a remote copy application as being a distributed system where two nodes work together to complete a single process but are separated by an intermediary network.

The same command/response protocols and operations that are used for local SCSI storage are also used for remote copy operations. Both forwarding and receiving controllers assume both initiator and target roles. The sections that follow explain the various SCSI roles assumed by forwarding and receiving controllers in a remote copy system.

Forwarding Controller as SCSI Target

All actions of the remote copy system begin with a forwarding controller receiving an incoming SCSI command from a host initiator. Obviously, this means the forwarding controller has to function as a SCSI target. If the incoming command is a WRITE command, the remote copy system is invoked, and the process of forwarding the command starts.

This forwarding controller traditionally resides in a disk subsystem, but it can also be located in a network device such as a switch, router, virtualization engine, or specialized remote copy appliance.

In its role as a SCSI target, the forwarding controller has to interact with the host initiator as a good citizen of the SAN, providing all necessary SCSI command responses, which are often referred to as acknowledgments in the context of remote copy systems.

Forwarding Controller as SCSI Initiator

Regardless of the type of SCSI command (READ, WRITE, or STATUS) sent by the host initiator, it is passed to its intended local LU for processing. If it is a WRITE command, the forwarding controller makes a copy of the command and then prepares to forward it to a receiving controller on the other side of the intermediary network.

The forwarding controller assumes the role of a SCSI initiator when forwarding commands to the receiving controller. It changes the target/LUN address of the original SCSI CDB and assigns the appropriate target LUN address that has been configured for secondary or tertiary storage in the remote copy system. It then reissues the command to the receiving controller.

Figure 10-2 shows how the forwarding controller processes WRITE I/O commands.

Figure 10-2. A Forwarding Controller Processes WRITE Commands

Receiving Controller in Dual Modes: SCSI Target and Initiator

The receiving controller assumes the role of a target in relation to the forwarding controller. It is assigned a target/LUN ID that is used by the forwarding controller initiator.

Upon receiving the forwarded command, the receiving controller is responsible for delivering the command to storage at the secondary site. That means it switches roles to act as an initiator. If the receiving controller is embedded in a disk subsystem, it only has to write data to memory cache or to disk storage. If the remote copy controller is external to a storage subsystem, it assumes the role of an initiator in the SANs and generates an I/O command for storage in a SAN.

In general, the receiving controller "owns" the storage address space on secondary or tertiary storage until the data needs to be used for running actual applications. At that time, a local system takes over operations and ownership of the storage.

When the command has been executed to completion (successfully or not) with secondary or tertiary storage, a SCSI command response (acknowledgment) is sent to the forwarding controller.

Figure 10-3 shows the various SCSI roles assumed by both forwarding and receiving controllers in a remote copy system.

Figure 10-3. SCSI Roles of Both Forwarding and Receiving Controllers in a Remote Copy System

Specialized Networking Equipment

The remote copy application may incorporate specialized networking equipment that improves the transmission of forwarded commands over the intermediary network. For instance, a specialized networking system equipped with large buffer memory and/or sophisticated error recovery technology could improve the reliability and recoverability of transmitted commands.

Such networking equipment or the forwarding controller may add transfer information such as parity calculations, sequence numbers, time stamps, or some other type of data to the forwarded command so that the receiving controller can correct transmission errors and avoid write-order violations.

Performance Implications of Remote Copy

The forwarding controller is responsible for acknowledging I/Os from host system initiators. This little detail of SCSI processing turns out to have an enormous impact on application performance and the budget allocated for business continuity operations.

Application performance can be constrained by the performance of the system's I/O channel. This is particularly noticeable with transaction processing applications that are sensitive to I/O latency. If acknowledgments are not returned quickly enough from storage, the application must constantly wait before sending the next I/O. As a result, the I/O channel becomes a bottleneck for application performance.

The challenge where remote copy is concerned is balancing the desire to maintain exact, synchronous mirror images of data on remotely located secondary or tertiary storage with the need to keep high-throughout applications running at an acceptable speed.

Of course, if money were not an issue, this challenge would not be so terrible. A company would simply use an intermediary network with sufficient bandwidth to support the amount of I/O write traffic generated by the application. But money usually is an issue. Despite the advancements made in recent years in reducing the costs of bandwidth, it still costs a great deal of money to support the number of write I/Os from a high-throughput application.

Latency of Light Propagation

Sometimes people mention the speed of light as a limiting factor for remote copy performance. While it certainly sounds great to say the words "speed of light," the speed isn't the problem. The latency related to the propagation of light signals through fiber-optic cables is the issue.

Light signals propagate in fiber cables at a rate of 5 microseconds per kilometer, or 8 microseconds per mile. You can use these numbers to calculate the latency of remote copy transmissions over fiber-optic links. For instance, the latency of sending a signal over a 20-mile piece of fiber is 160 microseconds.

This latency takes only the physics of light propagation into account. When you look at the complete picture, there are several other latency contributions to consider. These include the conversion of light to electrons on both ends of the connection, the interpretation and execution of the SCSI commands at the remote location, and any error corrections/retransmission processing that takes place, especially the impact of suspending command execution while waiting for late or missing transmissions in order to avoid violating data consistency requirements.

The Four-Stroke SCSI Write Process

Another issue to consider when analyzing remote copy performance is the need to transmit four signals across network cabling for SCSI WRITE processing. SCSI initiators send an initial command to an LU that indicates the intent of the write. The LU then responds that it can accept the transfer with a transfer-ready response. Then the initiator sends the data to the LU, which is written to cache or disk media, before acknowledging the second SCSI command.

Altogether that means four signals for each command need to be sent over the intermediary network. Using the 20-mile distance suggested earlier, we see that a single forwarded command has an associated remote copy command transmission overhead of 80 miles, or 640 microseconds. This is not insignificant compared to the I/O rates of high-throughput transaction processing systems. Figure 10-4 illustrates the four-stroke SCSI write process.

Figure 10-4. Four SCSI Signals for Each Write Operation

Synchronous, Asynchronous, and Semisynchronous Operating Modes

As it turns out, it is not always practical to maintain synchronization between primary and secondary or tertiary storage. For that reason, remote copy applications provide three different modes of operation:

• Synchronous

• Asynchronous

• Semisynchronous

Organizations must find a balance between the requirements of data consistency, distance between storage sites, and cost. It's not easy. Most companies find themselves wishing they could have synchronization of data for guaranteed data consistency, but they are unable to afford the cost.

Synchronous Mode

Synchronous mode works as outlined in the preceding descriptions of remote copy functions. The forwarding controller does not respond to the initial SCSI WRITE command from the host until it receives a response from the receiving controller that the write has been executed successfully. This is obviously the slowest, most conservative process.

Storage companies like to split hairs about whether the data is written to cache memory or to disk storage in synchronous mode operations. Obviously there is some associated risk that data written to cache memory might not ever make it to disk. For most companies it is a moot point because the cost of bandwidth for synchronous operations is too high.

Figure 10-5 illustrates the sequence of steps used with synchronous remote copy operations.

Figure 10-5. Sequence of I/Os and Responses in Synchronous Mode

Asynchronous Mode

Asynchronous mode allows the forwarding controller to acknowledge the completion of the WRITE command, after it is acknowledged by primary, local storage and without having to receive an acknowledgment from the receiving controller.

High-throughput applications are virtually unaffected by asynchronous mode operations. The downside, of course, is that there is no way of knowing what data transmissions have succeeded at the remote site at any given time. This makes recovering from remote communication failures slightly more complicated because more pending remote commands need to be tracked.

The forwarding and receiving controllers exchange all the normal SCSI commands and responses, even though the forwarding controller "spoofs" the host initiator with an early response for its original WRITE command. The forwarding controller retains forwarded commands until a response is received for them in order to recover from possible communication failures. Some remote copy products have a limit to the number of unacknowledged commands that can be pending. They will slow or stop responses from the forwarding controller to the host initiator in order to avoid getting too far behind in the process. The section "Link Failure Mode" discusses this further.

Figure 10-6 shows the sequence of steps used in asynchronous remote copy operations.

Figure 10-6. Sequence of I/Os and Responses in Asynchronous Mode

Semisynchronous Mode

Semisynchronous mode is a way to get a head start on a single subsequent WRITE command by assuming a response will be returned from the receiving controller for the pending command. The forwarding controller issues a response to the host initiator immediately upon receiving a response from the primary storage LU, as long as there are no pending I/Os in the remote copy system.

Link Failure Mode

When the forwarding controller loses communications with the receiving controller, the entire remote copy application is at risk. Different vendors have different ways of dealing with this problem. One way is to stop writes altogether. Another approach is to use local disk storage and create a remote copy cache that functions as a proxy remote site. The remote copy application tracks the data updated in the cache, and when the remote link is reestablished, the application sends the cached data to the remote site.

Cache data is marked using a storage block space bitmap. In other words, an index address space is created that has a single bit representing each block of storage. As the data is written to the cache, its corresponding index bit is turned on, indicating this block needs to be written to remote storage later. Bitmaps are used by other storage applications as well, including copy-on-write processes for backup, which are discussed in Chapter 13.

One of the interesting aspects of using a bitmap remote copy cache involves newer writes to cache that supersede previous writes on the same block. As it turns out, there is no need to send both writes, only the most recent one, saving time and bandwidth. Hotspot writes can be coalesced this way to reduce the amount of traffic sent over the intermediary network. In fact, this concept has been applied in some remote copy products that temporarily store writes in a cache and then send them as a group, as opposed to sending each write at a time. This technique would be effective in reducing remote copy bandwidth for applications like databases that have "hot spots" with repeated updates to the same blocks in rapid succession.