The IO Path

The I/O Path

Storage data is typically created by an application running on a system processor and then is transferred over the I/O path to its location on storage media. The remainder of this chapter describes the various hardware and software elements that storage data encounters on the I/O path. We'll begin by breaking down the entire end-to-end path and then take a second look at some of the more interesting aspects of host system software elements.

The I/O path can be conceived of as a set of distinct hardware and software elements, each with its own role, that acts on storage data as it moves from point to point along the I/O path.

Table 3-1 shows a generic "top-down" listing of the various hardware and software elements that comprise the I/O path.

Table 3-1. Hardware and Software Elements in the I/O Path

Component	Hardware	Software
Host system components	Processor Memory Memory bus I/O bus Network interface	Application File system Volume manager Multipathing driver HBA storage driver HBA network driver
Network components	Network media Port hardware Buffer system Switch core	Storage application Routing logic Flow control Virtual network
Storage subsystem components	Network interface Storage controller Cache memory Subsystem bus/network Storage device Storage media	Target/logical unit number (LUN) mapping Virtual disk manager

Host System I/O Path Elements

Host systems have many key software and hardware I/O path elements. Some of the elements, such as the system processor and memory, are used by virtually all functions in the system. Others, such as the file system and volume manager, are dedicated to storage I/O tasks.

Core System Hardware: Processors, Memory, and Memory Bus

Most of the action in a system happens within close proximity to the system CPU(s). This is where the software I/O path elements such as file systems, volume management software, multipathing drivers, and HBA device drivers run.

There are three core hardware system components of interest to storage networking:

• System processors

• System memory

• System memory bus

Figure 3-4 represents the three hardware inner-system components: processor, memory, and memory bus.

Figure 3-4. Processor, Memory, and Memory Bus

System Processors

The I/O path starts in a system processor when an application creates data that it needs to store. While many computer designs have a single CPU, many systems can use two, four, eight, or more to increase system performance. The number of processors does not materially change the I/O path, but it certainly affects the performance of system software components along the way.

System Memory

System memory chips are high-speed semiconductor devices having transfer rates in the range of nanoseconds. They are the fastest storage devices in a system and are typically used as cache or buffers for temporary data storage. The fast RAM used for system memory is volatile storage, which means the data is lost if the power is removed from the memory device. In general, it is a good idea to make sure there is plenty of memory in systems, especially those providing network services, such as NAS servers.

The System Memory Bus

Between the system and system memory is a high-speed bus for transmitting data from memory to the CPU and back. The system memory bus is not much of a variable when it comes to storage networking processes; it usually cannot be upgraded, expanded, or replaced.

System I/O Hardware: The I/O Bus and Host Bus Adapters

Systems communicate with storage through their I/O bus and an attached HBA. Storage data being transmitted from a system to storage is first transmitted from system memory over the system memory bus to the I/O bus to an HBA and then out to a storage network, as shown in Figure 3-5. A bridge controller chip manages the interface between the system memory bus and the I/O bus.

Figure 3-5. The Complete System Hardware I/O Path Picture

The I/O Bus

The I/O bus is a way to attach a variety of peripherals to a system. Like the system memory bus, the system I/O bus cannot be changed or upgraded. Unlike the system memory bus, which normally has plenty of bandwidth for its work, the system I/O bus has lower performance capabilities that limit the total amount of data that can be transferred at any given time through the system. When calculating storage I/O workloads, remember that storage data crosses the I/O bus both on its way in and out of the system.

NOTE

Eventually I/O bus technology will probably be replaced with some sort of I/O networking technology, but I'm not holding my breath considering the delays and disappointments that have occurred in the development of InfiniBand technology. Tired of using convoluted terms like "I/O bus/network," I'm more than happy, for now, simply to refer to I/O connecting stuff in a system as the I/O bus until something else comes along that actually has more market traction.

Host Bus Adapters

The last piece of system hardware in the I/O path is the HBA. It performs the same types of actions that network interface cards do in other types of networks, except that they also have a storing level component for managing the exchange of storage information. From a purist's networking perspective, the HBA is a specialized network interface card (NIC) with application level functions built in. The term storage NIC is sometimes used to refer to an HBA for iSCSI SANs.

Host System Storage Software: File Systems, Volume Management, Multipathing Drivers, and HBA Device Drivers

Although they are often sold as separate software products, file systems, volume managers, multipathing software, and HBA device drivers work transparently together as the software elements that control the I/O path.

NOTE

The high-level overview of the I/O path that follows completely ignores the most important software in the machinethe operating system. Obviously, keeping the OS out of the discussion is just not right! Unfortunately, adding the operating system to high-level I/O path discussions clarifies nothing because the OS has no specific defined storage I/O role. Instead, it works closely with all the other host software elements in the I/O path. The section "The Operating System's Role in the I/O Path" near the end of this chapter takes a look at the role the OS has in storage I/O processing.

File System

File systems are filing entities, managing the placement of data in storage address spaces. The I/O path starts when an application makes a file I/O request (also called a file system call) to a file system. The file I/O request specifies the name of the file as well as optionally indicating which bytes within the file to work with. File systems do a number of things, but where the I/O path is concerned, their job is to translate relative byte locations within a file to block addresses in a storage address space. Figure 3-6 is similar to Figure 2-4 in the previous chapter, but Figure 3-6 shows a file system receiving a file I/O request for a range of bytes within a file and converting it to a request for blocks of data.

Figure 3-6. A File System Converts a Byte-Range Data Location to Its Associated Block Addresses

While most file systems are sold as part of operating system software, they can also be purchased separately or as part of a storage network solution. Systems can have multiple different file systems running concurrently, accessing data on different storage address spaces. The characteristics of the file systems installed in a system have an enormous impact on the system's network storage capabilities.

When the file system is done processing the file I/O request, it passes it as a block I/O request to the next stop in the volume management processthe volume manageras shown in Figure 3-7.

Figure 3-7. A File I/O Request Is Processed by the File System and Is Passed Down the I/O Path to the Volume Management Function as Multiple Block I/O Requests

Volume Managers

Volume managers are responsible for creating storage volumes by manipulating storage address spaces. These volumes are typically identified by users and applications as drive letters such as C:\ or D:\. Like subsystem controllers and storage virtualization products that manipulate storage address spaces, volume managers aggregate, subdivide, and mirror storage resources in the SAN. Volume managers are optional, but all systems have a program that creates disk partitions, such as DOS's simple FDISK function.

The volume manager creates one or more downstream (toward storage media) block I/O requests from a single upstream I/O request that it receives from the file system. Figure 3-8 shows a volume manager creating three downstream block I/Os from a single upstream I/O created by a file system.

Figure 3-8. A Volume Manager Creates Three Downstream Block I/Os for a Block I/O Request It Receives from a File System

NOTE

The terms upstream and downstream are used in this book to refer to the general direction of an I/O process. Upstream refers to a process where a command or data is being transferred to an initiator. Downstream refers to a direction away from an initiator.

When the storage I/O request has been processed by the volume manager, the next stop along the I/O path is multipathing software drivers.

Multipathing Drivers

High availability for storage I/O dictates redundant paths, including redundant host initiators in the SAN. Software that manages the configuration and use of redundant paths in a SAN is called multipathing software. Multipathing is usually implemented as an upper-layer storing device driver that determines which connections are used for storage transmissions. Chapter 11, "Connection Redundancy in Storage Networks and Dynamic Multipathing," discusses this important topic in detail.

NOTE

For the most part, multipathing is primarily used to change the HBA that is used, regardless of the nature of the connection failure. Multipathing uses storing level addresses, particularly the host initiator and the storage target addresses.

The network path used for storage I/O is also part of the I/O path. However, the network path is determined by routing algorithms in network switches and routers and is independent of the storage connection selected by multipathing software. Multipathing and network routing work together but function like ships passing in the night that have little knowledge of the other's mission. If you want to improve the world a tiny bit, think of ways to collaborate timeout values between the storing and connecting worlds so that network switches can correct link errors before SCSI errors force multipathing changes.

HBA Storing Drivers

The HBA's storing drivers are usually referred to by their corresponding technology, such as "the SCSI-layer driver." After the block I/O addresses have been identified by the file system, translated by the volume manager, and passed to the appropriate HBA by multipathing software, a storage-level HBA driver formulates the storage command or data transfer payload that will be transmitted to the target device or subsystem.

HBA Network Drivers

The last piece of host software to see data transmission on the way out of the systemand the first to see it on the way back inis the HBA's network driver. This software participates in the actual transmission of signals over the network or bus.

The Host Software Stack

It's possible to think of storage software as a stack of functions, as shown in Figure 3-9. The top-left corner of Figure 3-9 shows the top of the stack, starting with an application that creates and uses data. The next level down is a file system that determines where the data is stored and how it is accessed. Next comes the volume manager, which creates storage address spaces in the host using downstream device and subsystem storage. Finally, the bottom of the stack is the device driver level, which has an optional multipathing component as well as required storage and network components.

Figure 3-9. The Storage Software Stack

The Complete Host System I/O Path

Figure 3-10 shows the combined picture of host I/O path elements. The top-left corner of Figure 3-10 shows the hardware elements that surround the processor, including memory and the system memory bus. The top-right corner of the figure shows the software elements that run in the system CPU and use memory resources to accomplish their tasks. The remainder of the figure is the same as Figure 3-5, except for the dotted line between the device driver software on the far upper-right corner and the host bus adapter at the bottom. Device driver software runs in the system processor, but it interacts with the host bus adapter using interfaces provided by the operating system.

Figure 3-10. Combined Host System Hardware and Software Elements of the I/O Path

Network Systems and Components in the I/O Path

Networking products provide the connecting function in a storage network and obviously have key roles in the I/O path. As the technology continues to develop and mature, many expect networking systems to assume larger roles that will include enhanced management capabilities as well as storing and filing functionality.

Network Hardware

Networking technology has both hardware and software components; we'll look at network hardware first. The network hardware covered in this section is as follows:

• Network media

• Ports

• Buffer memory

• Switching core

Network Media

While networking systems may see expanded roles in the I/O path, the media itself will not. As an inert, inactive element, the most important things media can do for the path are to increase transmission accuracy, extend distance, and improve security (with optical media).

Ports

The I/O path goes through networking equipment (switch, director, router, etc.) by way of its ports. Network ports may have specific definitions and roles or may be available for a variety of functions. Ports in network switches tend to be very fast and are designed to forward storage transmissions with a minimal number of delays. Ports in other types of devices, such as SAN appliances, might use HBAs as opposed to switch ports.

Buffer Memory

Storage data that is forward by a switch, router, or bridgeor otherwise processed by a network systemis temporarily placed in buffer memory before being forwarded to the storage target.

On the surface, buffer memory seems like a simple matter, but the architecture and implementation of queues in buffer memory can make significant differences in storage network capabilities and management. A single buffer in a switch can impact many different I/O paths for many different pairs of targets and initiators.

NOTE

One could say that buffer memory is just a part of a switch's port and is not worth singling out as a separate I/O path element. But that would be slighting buffers unfairly. Buffers and queuing in network switches are a subsystem within the switch, implemented in association with ports.

Switching Core

Storage network switches and directors (large, high-availability storage switches) are among the most important products in the I/O path. A switch's performance and availability capabilities depend on the design of its core switching technology, which is the primary I/O path element in a switch.

Network Software

As storage networks grow in size and complexity, networking software becomes increasingly important as a way to safely and accurately configure, change, and manage a storage network. Some of the network software elements directly involved in the I/O path are routing logic, flow control, virtual networking, and storing applications.

Routing Logic

The implementation of routing algorithms is among the most important part of any network, including storage networks. In addition to simply determining how to route storage I/O traffic from ingress to egress ports, routing algorithms can also determine the prioritization of traffic if failures occur in the network. Prioritization of multiple I/O paths is highly desirable in large storage networks.

Flow Control

Flow control is the ability to adjust the data rate for a transmitting network node. It does not impact the addressing, commands, or payloads of storage I/O, but instead changes the frequency at which transmissions are made on the I/O path. Considering how heavy and irregular storage I/O traffic can be, it may be desirable at times to throttle back certain nodes and reduce network congestion. As a rule of thumb, storage networks work best when unnecessary data retransmissions can be held to a minimum.

NOTE

This last sentence would be a "duh!" for all networks. By the way, flow control can also be implemented in the HBA and network device driver code, as it is with Transmission Control Protocol (TCP). However, in Fibre Channel SANs, most connections are Class 3 and do not support end-to-end flow controlwhich means if meaningful flow control is going to happen, switches need to be involved.

Virtual Networking

Virtual networking technologies such as virtual LANs (VLANs) and virtual SANs (VSANs) are closely related to routing in that they can also be used to determine where to forward incoming frames through a switch. However, virtual networking is much more than a different way to implement frame forwarding; among its virtues are strong I/O path segregation and the ability to assign quality of service (QoS) levels to I/O paths.

Storage Applications

There is considerable potential for running storing or filing applications within a switch or other storage network system. If so, these applications would affect the I/O path in much the same way as the host software processes discussed previously. Additional hardware components, such as memory, would also be required to support application processes.

For example, a network switch could provide a security function that would restrict access to downstream subsystems without proper credentials. Another storage application that could run in a network system is storage virtualization, the subject of Chapter 12, "Storage Virtualization: The Power in Volume Management Software and SAN Virtualization Systems."

The Complete Path Through Network Systems and Components

Figure 3-11 illustrates network I/O path elements. On the upper-left corner of this figure is the network media. Below that is the network port and the buffer memory, which is used to temporarily hold storage transfers that are en route. The bottom layer of this drawing shows the switching core elements, including the core hardware on the left and the primary supporting software functions of routing, flow control, and virtual networking on the right. Storage applications are also shown on the right, although these are not likely to run in the core function of a switch, but rather through an add-on module of some sort that interfaces with the switch.

Figure 3-11. Network I/O Path Elements

Storage Subsystem Hardware

The last part of the I/O path left to examine is within storage subsystems. While this topic is the subject of Chapter 5, it needs to be discussed briefly here to complete the overall picture of the I/O process and path.

Storage subsystem hardware has six basic I/O path elements:

• Network ports

• Storage controller

• Cache memory

• Internal interconnect

• Storage devices

• Storage media

Subsystem Network Ports

Storage subsystems can have one or more ports with multiple virtual disks that are made accessible, or exported, through each port. Subsystem ports can be implemented as circuits built from chip components, but they are more likely to use HBAs. After storage transmission is read by the subsystem port, it is copied to the storage controller for further processing.

Storage Controller

Controllers in storage subsystems may be implemented with off-the-shelf products or with unique, proprietary designs. The subsystem controller interprets incoming storage I/O commands and then issues downstream commands to storage devices inside the subsystem.

Cache Memory

Memory is much faster than magnetic storage, such as disk and tape drives. As a result, storage subsystems often have cache memory to improve system performance by decreasing I/O response times. The subsystem controller is responsible for determining what data to place in the cache, what data to write to disks, and what data to delete from the cache.

Internal Interconnect

All subsystems have some type of interconnect, such as a storage bus or private network, for transmitting storage data from the subsystem controller to storage devices. High-availability subsystems provide redundant internal connections between the subsystem controller and the subsystem's internal storage devices.

Storage Device

The basic building block of network storage is the disk drive. It has its own connection ports and a controller for internal operations and logic. Although far fewer tape drives than disk drives are installed, they are used almost universally as a storage device for backup and recovery.

After a disk or tape drive receives a storage I/O command from the subsystem controller, it performs the action requested and reads or writes data through an internal connection called the read/write channel. The read/write channel connects the device's controller with the recording heads that are ultimately responsible for creating and detecting the magnetic signals on metallic media that we refer to as stored data.

Storage Media

The I/O path ends at the storage media, where data is persistently stored. When writing data, the recording heads in a storage device induce a magnetic signal to be imprinted on the moving magnetic surface of storage media. Data resides on storage media as magnetically imprinted "1 and 0 bits." When reading data, the signals are detected as voltage changes as the media moves past the head.

The Complete View of Storage Subsystem I/O Path Elements

Figure 3-12 shows the storage subsystem I/O path elements discussed earlier. I/O operations begin with the connection port on the left of Figure 3-12 and then are processed by the subsystem controller. The subsystem's cache might be used, depending on I/O details and the design of the subsystem. The controller creates one or more downstream I/Os that are transmitted across the subsystem's interconnect to storage devices, and from there the device transfers them across its read/write channel to media where the data is stored.

Figure 3-12. Storage Subsystem I/O Path Elements

The Operating System's Role in the I/O Path

NOTE

The conceptual I/O path model shown in this chapter is fairly compact, allowing people to easily understand how storage data transmissions work. More importantly, the conceptual model allows people to design effective storage network architectures.

But that does not mean it answers questions about how things really work in host software. The unfortunate fact is, relatively few people understand the OS inner workings of the I/O path. Experts in the field, whom most of us would consider "computing geniuses," spend their working lives in and around this topic and are often perplexed by the complexity involved.

Nonetheless, I believe some insight into and appreciation of this greater complexity is useful. Readers are advised that storage I/O, like sausage and legislation, may be best understood by the results, rather than the makings.

If the CPU is considered the brains of the system, the operating system is its consciousness. As the controlling logic that manages access to the CPU, memory, and communication resources and schedules all work in the system, the operating system has a profound impact on the storage I/O capabilities of a system.

Operating system processes separate their work into two primary processing environments:

• User space (also referred to as running in protected mode on Intel-compatible processors)

• Kernel space (also referred to as running in real mode on Intel-compatible processors)

User space is the system environment provided by the operating system for all processes that are not involved in running the system. This includes application software as well as many system functions. User space processes do not access system hardware or firmware, but rely on the operating system kernel for these actions. Each user space application is given its own context (virtual address space and state) by the operating system. Storage networking software that runs primarily in user space includes such products as backup/recovery and storage resource management.

Kernel space is the processing environment where the operating system manages system hardware and firmware resources. Kernel processes are sometimes referred to as primitives. There is typically a single context for all kernel space functions, and kernel processes need to start and finish quickly. File systems and device drivers run primarily in kernel space as high-priority, high-performance system subroutines.

Operating systems have a variety of interfaces that applications and system software use to pass control messages and data from one process to another. Transitions between user space and kernel space take a lot of CPU resources, and vigorous attempts are made to minimize them.

File System Integration

File systems are the transition operator between user space file I/O requests and kernel space block I/O storage services. In other words, file systems are smack in the middle of many of the user space/kernel space transitions that go on in a system. Keeping in mind that such transitions are relatively expensive and that attempts are made to minimize them, it turns out that the file system and operating system have to be tightly integrated.

The most intricate interfaces in a system are those between the operating system kernel and the file system(s). A single storage I/O request from an application may generate many exchanges between the kernel and the file system, some in user space and some in kernel space, but all require fast code execution without errors or bugs. A great deal of operating system and file system development efforts are spent fine-tuning and testing file system/kernel interactions.

The I/O Operation

It's convenient to talk about the "I/O operation" as a compartmentalized process with a consistent data structure that is passed from process to process with clearly defined abstraction boundaries between them. In fact, nothing could be further from the truth. I/O operations are much more like spaghetti structures with many different processes that access data as lists of addresses on disk or pointers to memory addresses.

NOTE

In fact, the OS gets involved immediately with every I/O operation. In the I/O path model discussed earlier, the first interaction in the I/O path was between the application and the file system. However, this was a bit misleading for the sake of brevity. The application actually makes the file system call to the OS, which passes the call to the file system.

Installable File Systems

Historically, file systems were implemented as part of the operating system, but over time most operating systems externalized subsets of their file system functions through programmatic interfaces. File systems that use these interfaces are called installable file systems (IFSs). Installable file systems allow existing operating systems to support new storage devices when they are introduced to the market, such as when DVD-RAM drives were introduced. More importantly for storage networking, installable file systems can provide higher levels of data availability through clustered or distributed file systems.

An example of an IFS is a NAS client. NAS access depends on a special type of file system technology loaded on client systems that redirects file I/O from client systems over a network to a remote file system running on a NAS server.

Figure 3-13 shows how a NAS IFS in a client redirects file I/O over a network to a remote system.

Figure 3-13. A File I/O Request Is Redirected by a NAS Client IFS to a Remote NAS Server

IFS implementations may be used for a wide variety of purposes. Where storage networks are concerned, these tend to involve methods of sharing access to data among clustered or distributed systems.