Communications Capabilities

The capabilities defined in this section provide integration at the communications level.

Request/Reply

The Request/Reply capability facilitates synchronous communication between applications. In request/reply interactions, the source application sends a request to the target and then performs no other processing until the reply is received.

Request/Reply does not normally use queuing because the requestor is waiting for a reply. Instead, a timeout parameter is set. If the reply is not received in a certain amount of time, the requester resends the request.

Request/Reply also uses other capabilities in the environment to provide its functionality. For example, the Directory capability can resolve addresses for the Request/Reply capability.

To enable request/reply interaction between two applications, a common data format is required as well as common data and process semantics.

Connection Management

When a request/reply interaction takes place, the Connection Management capability establishes a logical connection between the requesting and the replying applications. Connection Management assigns a connection ID to both applications and then returns the reply to the application that sent the request.

Message Routing

The Message Routing capability controls the routing of messages from source to target applications. Message routing is a quite simple problem when it entails moving the data from Point A to Point B. In other conditions, however, the routing process can be quite elaborate — for example, when moving the data from Point A to a hub, where the message may be forwarded to Points B, C, or D, based on a variable such as time of day, server load, or message content.

The message itself is a formal construct containing the data. You can think of the message format as an envelope, with the contents of the envelope being routed based on the addressing information that is listed on the outside of that envelope. In practice, the addressing information is often placed in a message header and the content of the message then follows the header in the message's data structure.

The routing functionality is built into the message hub, which inspects each message in sequence, determines where to send the message, and then sends it to the target system.

Addressing

Your application integration environment cannot route messages correctly unless it can determine where to send the messages. The Addressing capability maintains the information about where to route the message. The environment can determine the location of the target system(s) through either direct or indirect addressing.

Direct Addressing

Direct addressing is used when the source application specifies the target application for communication. The source application may either specify a network address or a name, that the Addressing capability in turn resolves to a network address, through a call to the Directory capability.

Indirect Addressing

Indirect addressing occurs when the source application does not specify a target application. Instead, the Addressing capability determines the correct target application after the message has left the source application.

There are two forms of indirect addressing: publish/subscribe addressing and content-based routing.

Publish/Subscribe Addressing

In publish/subscribe addressing, the source application sends out the message without indicating the target application; the identity of the target application may not even be known to the source application or its developer. Instead of a destination address, the source application identifies each message by providing a topic name.

Independent of the activities of the source application, potential target applications register with the Addressing capability when they are initialized. During this registration, each target application indicates that it is interested in receiving all messages on a specified topic.

Publish/subscribe addressing provides a useful anonymity to sources and targets, meaning that applications can be added or removed at any time. At run time, the messages are published by the source application(s) with topic names, and the Addressing capability builds a routing table automatically to route each message to the target application(s) that had previously subscribed to messages about the topic. Published messages are often kept in a data store for access by subscribers.

Content-Based Routing

Content-based routing is closely related to publish/subscribe addressing. In content-based routing, the source application again does not specify the target application's identity. However, content-based routing differs from publish/subscribe addressing in that a developer builds a routing table that specifies rules to determine the correct match between a particular message and a target application. For example, the table may ensure that all orders for a certain product go to the factory that makes that product.

Message Forwarding

Sometimes a message must be passed through several applications before a response to the original request can be completed. The Message Forwarding capability handles such serial messages.

The forwarding capability can also serve another purpose. In some integration implementations, several intermediaries are needed to provide message management and monitoring capabilities. In such cases, Message Forwarding moves the data between each of the intermediaries, while keeping track of the required sequence.

Message Delivery

The Message Delivery capability determines how messages pass from source application to target application. Message Delivery occurs through one of two mechanisms: multicasting or single casting.

Multicasting

Message delivery through multicasting involves delivering the same message to all (or nearly all) of the potential target applications and letting them decide whether to keep the message or not.

The multicasting approach may seem rather wasteful of bandwidth. In reality, however, most local networks are Ethernet-based, which works according to a similar principle, so the multicasting approach does not lead to excess traffic on these networks. In fact for transmission over local area networks (LANs), multicasting is often the most efficient form of message delivery.

Of course, the vast majority of networks consist of multiple LANs. In situations where the message is intended for addresses on other LANs, gateways are used to forward the multicast message to the LANs where the remote addresses reside. Most integration scenarios involve the bridging of several LANs with message delivery gateways.

Single Casting

Message delivery through single casting involves delivering each message to specific targets as identified by the Addressing capability. Single casting is not always the most efficient way of sending messages, because addresses have to be resolved before they can be sent to each recipient, which can slow performance. However, single casting is very useful in some circumstances; for example, when a number of intermediaries perform routing, filtering, and forwarding functions on the message.

Message Queuing

In application integration, it is often important to provide guaranteed delivery of each message to ensure that the message reaches its recipient. It is also important that each message is delivered once and only once to its destination. Otherwise, it would be easily possible, for example, for a customer to receive multiple shipments of goods when he or she placed only one order. Finally, it is important that the messages arrive at their destination in the order in which they were sent. This ensures that, for example, an order correction does not arrive before the order that it is correcting.

The implementation of reliable messaging depends on the Message Queuing capability. This capability stores each message in a queue until it can be successfully delivered.

This queue can be implemented in memory or on disk. Implementing this queue in memory can maintain high performance, but at the risk of losing messages if the server fails. Disk implementations trade off higher reliability for slower performance because disk I/O is required every time the message metadata has to be updated to reflect completion of another step in the guaranteed delivery process.

When a message is not delivered successfully, it remains in a queue, and the Message Delivery capability attempts to redeliver the message. After attempting delivery a set number of times at a set interval (as defined by an administrator), the Message Delivery capability places any messages that cannot be delivered in a dead letter queue. Administrators can inspect the dead letter queue, and correct a message addressing header or message content to ensure successful retransmission to the intended destination.

At this point, a message may also have to be sent to the source application to indicate this failure. Or, a message may be sent to an administrator who can help determine the source of the addressing error.

Message Instrumentation

Although messages are sent in a specific order, they can be received in any order. The Message Instrumentation capability helps ensure that the messages are processed in the order that was originally intended.

The correct order can be ensured by using sequence numbers on the messages themselves. In this case, Message Instrumentation adds the sequence numbers and accepts messages only if the previous message has been correctly received. It may also add the name of each intermediate hub to every message along with a time stamp as it reaches each hub. This additional information enables you to create historical reports on messages and may assist in resolving any performance issues.

Message Correlation

In an asynchronous interaction, there is no logical connection between the sending and receiving applications. When the receiving application replies to the initial transmission, the receiving application has to specify the proper application name or network address along with the reply message.

Because an application may send several messages asynchronously before receiving a reply for any of them, the application that sends asynchronous messages uses the Message Correlation capability to match up or correlate each reply with the corresponding transmission. Each time a message is sent or received, the Message Correlation capability is called so that it can append a tracking number to the message header. In terms of functionality, this mechanism is very similar to the Connection Management capability used in synchronous messaging.

Serialization/Deserialization

Data often exists in a hierarchical format in applications. However, as it travels between applications, data must be converted into a flat structure so that it can be sent over the network. The Serialization capability does this conversion. Of course, after the data arrives at its destination, it must also be converted from a flat structure back into its original hierarchical structure. The Deserialization capability does this conversion.

Transactional Delivery

Often, you must guarantee the delivery of messages. The Transactional Delivery capability groups together messages, sending and receiving them within a transaction. This means that such messages either are sent together in order (a committed transaction), or are not sent at all (an aborted transaction). Likewise, transactional messages are only read (removed) from a queue if and when the transaction is committed. Otherwise, they are returned from a queue and can be subsequently read during another transaction.

The Transactional Delivery capability ensures that transactions meet the following requirements, known collectively as ACID (Atomicity, Consistency, Isolation, and Durability), which are defined as follows:

• Atomicity refers to the need to complete all the parts of the transaction or none of them. For example, when a pick list is generated to take a book from the shelf, the inventory quantity for that book should be decreased.

• Consistency refers to the need to maintain internal consistency among related pieces of data. For example, in a funds transfer application in a bank, a $100 withdrawal from a customer's savings account should be completed successfully only if a corresponding $100 deposit is made in that customer's checking account.

• Isolation refers to the need to ensure that the activities within one transaction are not mixed with the actions of parallel transactions. For example, if five simultaneous requests for a particular book arrive and the warehouse has only four copies of the book in stock, the five transactions must each run in isolation so that the orders are processed properly. Otherwise, it might be possible for each of the five business transactions to see an inventory level of five books and then try to reduce the number to four.

• Durability refers to the need to ensure that the results of the transaction are changed only by another valid transaction. The results of the transaction should not be compromised by a power outage or system failure.

To assist in providing durability, your transactional delivery mechanism could employ a disk-based delivery method, where every message is written to permanent storage as it moves through the system. However, to provide durable transactions, the message must be committed to a persistent storage the moment the transaction is completed.

Note

To overcome system unavailability, you can use persistent messaging as an alternative to Transactional Delivery. Persistent messaging makes the message itself more resilient.

Transactional Delivery has a performance cost, because of the overhead involved in ensuring that delivered messages can be rolled back, or in resending messages after a delivery failure. However, the performance tradeoff is often necessary to provide additional guarantees during message delivery.

Batching/Unbatching

You may not always have continuous, fast, and reliable network connections throughout your enterprise. It can therefore be very useful to collect messages and then send them when the link is available. The Batching/Unbatching capability provides this functionality.

One of the disadvantages of a batching and unbatching approach to communications is that it can lead to unacceptable inconsistencies in data. For example, the same object can be deleted from a database in two separate locations, but there may be no realization that this has occurred until the batch processes take place later.

Encode/Decode

Different operating systems use different character encoding methods for representing text characters. For example, IBM mainframe applications use Extended Binary-Coded Decimal Interchange Code (EBCDIC) as a character encoding method, while other systems either use ASCII or Unicode. The Encode/Decode capability is responsible for ensuring that applications can still communicate, despite using different encoding methods.

Even when two systems use the same encoding method, there may be differences in the actual encoding due to the use of different code pages. These different code pages are necessary because many languages use different alphabets. Some vary in minor ways (for example, Spanish uses a character constructed by placing a tilde above the character "n"). Others are radically different (for example, the Romance and Germanic languages use the Latin or Roman alphabet, whereas languages such as Arabic, Hebrew, Japanese, and Korean use entirely different alphabets).

When communication capabilities receive the data, the Decode capability is invoked as necessary to convert the data to the encoding method and code page used by the components of the integration solution. This decoding ensures that integration functions such as transformation and routing can be properly performed. After the integration middleware is ready to transmit a message to the target system(s), the message may have to be recoded by using the encoding system and code page of the target system(s).

File Transfer

In some cases, rather than using request/reply or message-based communication, applications must communicate by exchanging files over the network. The File Transfer capability handles this process of moving or transmitting files between applications.

File Transfer can be useful in an environment when applications are not designed to communicate with other applications directly, but are designed to read, manipulate, and create files. You may also use the File Transfer capability in conjunction with the Batching/Unbatching capability to collect messages into a file, and then transfer that file at a particular time or date, or when a reliable communications link to the target system is available.

Although file transfer lacks some of the sophistication of other communication capabilities, it is still a widely used communication mechanism, due to the number of applications that read and write to files as their only way of communicating. One advantage of file transfer over messaging-based communications is that this type of communications usually is possible within a simpler infrastructure than messaging.