1.3 Message Passing versus Shared Memory

I l @ ve RuBoard

Our discussion of protocols has so far assumed information is transferred between peers that reside on different computers. This type of exchange is referred to as message passing. A different type of information exchange can occur when communicating peers have access to a region of shared memory. This section describes these two mechanisms and their tradeoffs.

Message passing exchanges bytestream and record-oriented data explicitly via the IPC mechanisms that we'll explore in Chapter 2. Application developers generally define the format and content of these messages, as well as the application protocol that participants must follow to exchange messages. This protocol also defines the number of possible participants in each exchange (e.g., point-to-point ( unicast ), multicast , or broadcast ) and how participants begin, conduct, and end a message-passing session.

The message-passing IPC mechanisms transfer data from one process or thread to another in messages across an IPC channel, as shown in Figure 1.3 (1). If a large amount of data is transferred, it may be fragmented and sent in a sequence of messages. If there's more than one process receiving the data, each message may be sent multiple times, once for each recipient. Many popular middleware architectures, such as RPC, CORBA, and message-oriented middleware (MOM), are based on a message-passing communication model internally.

Figure 1.3. Message Passing versus Shared Memory

Shared memory allows multiple processes on the same or different hosts to access and exchange data as though it were local to the address space of each process. When networked applications have data that must be viewed and/or manipulated by multiple processes, shared memory facilities may be a more efficient communication mechanism than message passing. Rather than define the method for transmitting information between entities, applications using native OS shared memory mechanisms must define how to locate and map the shared memory region(s) and the data structures that are placed in shared memory.

Shared memory can be divided into local and distributed variants, as described next .

Local shared memory allows processes on the same computer to have one or more shared memory regions mapped to different virtual address ranges. Two common shared memory mechanisms include

1. System V UNIX shared memory, wherein the shmget () system function creates a new region of shared memory or returns an existing one. A process attaches a shared memory region to its virtual address space with the shmat () system function.

2. Memory-mapped files, wherein all or part of a file can be mapped to an area of virtual memory that's shared across multiple processes. The contents of memory-mapped files can be flushed to persistent storage, which provides a convenient way to save and restore information across program execution.

In both mechanisms, the OS enables multiple processes to map shared memory regions into their address spaces, as shown in Figure 1.3 (2). All processes that map the shared memory region(s) can read from and write to the shared memory's contents directly.

Despite its flexibility, communicating between multiple processes via shared memory requires careful programming. For example, application developers must ensure that data being shared aren't corrupted by race conditions. The system-scope synchronization mechanisms described in Chapter 10 can enforce serialization to a shared memory region so that shared information is accessed in an orderly manner. Likewise, applications must be designed carefully if they store C++ objects in shared memory, as described in Sidebar 3.

Sidebar 3: C++ Objects and Shared Mernory

The C++ placement new operator can initialize C++ objects in shared memory. This feature works for concrete classes [Bja00] because each contains all operations necessary for its support. All methods in a concrete class are nonvirtual; that is, they are called directly rather than indirectly through a pointer-to-function. Many ACE wrapper facades are concrete types, for the reasons described in Section A.6 on page 255.

In contrast, abstract types that contain virtual methods are hard to program portably when placed in shared memory. Virtual methods are usually invoked using an indirect call through a table of function pointers (a vtable) located in the object's memory. A shared memory region may reside at a different virtual memory location in each process that maps it [BC94, Jor91]. Moreover, a C++ compiler/linker needn't locate vtables at the same address in separate processes. The vtable, as well as the functions it refers to, may therefore be mapped at different virtual addresses in each process, which guarantees problems at run time.

Distributed shared memory (DSM) is a programming abstraction that provides applications with an extension to OS virtual memory mechanisms [JM98]. Virtual memory provides a set of policies for fetching, placing, and replacing a set of pages on demand to give programs the illusion that they have a much larger address space than actually exists in physical memory. DSM extends the virtual memory concept across a network to enable transparent interprocess communication via data in global/shared memory. It represents a hybrid of two computing paradigms : shared memory multiprocessors and distributed systems [PTM97].

For example, there are hardware/software platforms that cluster multiple computers into one logical system in which memory is shared across the cluster. Participating applications communicate via distributed shared memory managed across the networked computers. DSM systems provide mechanisms for coordinating updates across processes and network nodes. This communication model often employs message-passing mechanisms in its lower layers to coordinate the exchange of data and to synchronize readers and writers.

Logging service Our networked logging service uses a message-passing IPC model implemented over TCP/IP. Message-passing IPC is usually more practical than developing a DSM mechanism and managing it via complex distributed cache coherency protocols. Since DSM is an advanced research topic that's not commonly used in practice, we don't consider it further in this book. Good overviews of DSM research appear in [NL91, PTM97].

I l @ ve RuBoard