Important Considerations for Application Integration
The specifics of your application integration environment depend on many things, including the complexity of both your business processes and your IT environment. Several considerations, however, are common to many application integration environments. This section discusses these issues.
| Note | For more details about the specific capabilities that are required for an application integration environment, see the Appendix, "Application Integration Capabilities." |
Using Synchronous and Asynchronous Communication
One important consideration in an application integration environment is how the applications communicate with each other. They may use synchronous or asynchronous communication, or most commonly, a combination of the two.
Often the applications themselves determine the best forms of communication to use. In situations where the application must wait for a reply before it continues processing, synchronous communication is often most appropriate. Asynchronous communication is normally appropriate in situations where the application can continue processing after it sends a message to the target application.
Just considering the needs of the application may be a too simplistic way of determining whether to use synchronous or asynchronous communication, however. Communication may be synchronous at the application level, for example, but the underlying architecture that enables this connectivity may be asynchronous.
Generally, a number of capabilities are required to support synchronous and asynchronous communication. Table 2.4 shows these capabilities.
| Form of communication | Capabilities required |
|---|---|
| Synchronous communication | Request/Reply |
| Asynchronous communication | Message Routing |
Asynchronous communication offers a potential performance advantage, because applications are not continually waiting for a response and then timing out when they do not receive one in a timely fashion. Asynchronous communication offers other performance benefits as well. Because asynchronous communication is connectionless, a continuous link is not maintained between each side of a data exchange. This arrangement can provide a small performance advantage over connection-based communications, and may also reduce the load on the network. However, any advantage may be offset, at least in part, by a need to match replies to initial messages at the application level. Using the Message Queuing capability to ensure message delivery can also diminish performance.
Although applications traditionally rely on synchronous communications, applications can maintain the same functionality while taking advantage of the benefits of asynchronous communications. In most cases, you should design your application integration environment to support both synchronous and asynchronous communication, because more often than not, both will be required. However, you should consider adopting an asynchronous model for communications whenever possible.
Increasing Automation
One of the most significant aims of an application integration environment is to reduce the amount of human involvement in integrating applications, and therefore to move increasingly towards fully automated application integration. Of course, in many cases this is not possible. However, it is likely that some of your organization's business processes can be more automated.
In particular, you may have many applications that enable you to perform business processes, but that require a series of manual steps. Manual steps might include interacting with a number of applications consecutively. As an example, if a staff member named Bob orders 100 widgets from the Manufacturing department, the business process might resemble the following.
-
Bob chooses the widgets in an online catalog.
-
Bob creates a purchase order for the widgets.
-
Bob sends the purchase order to his manager for approval.
-
Bob's manager approves the purchase and sends a copy of the approved purchase order to the Finance department. Finance stores the purchase order as a pending purchase.
-
Bob sends the purchase order to the Manufacturing department, copying Finance so they know the purchase has been made. Finance stores the purchase order as an outstanding purchase.
-
Manufacturing sends the widgets, updates its inventory, and invoices Finance for the purchase amount.
-
Finance transfers the money to the widgets manufacturing department.
The previous example uses a series of applications in a specific order. Much of this processing can be automated so that applications interact with each other. For example, by integrating the online catalog with the application creating the purchase order, you can save Bob a step and save the company some money. Actually, this type of process can be completely automated (with pauses for human approval). Some of the steps are sequential, but others can be performed in parallel (for example, Finance can store the purchase order as an outstanding purchase as Manufacturing sends the widgets). The order of events depends on the overall nature of the business process.
Straight-Through Processing (STP)
Some business transactions do not require any human intervention. The processing for such transactions is known as straight-through processing (STP). The goals of using STP include:
-
Improving customer service by reducing the opportunity for human errors and by speeding up the business process handoffs.
-
Reducing the cost of the business process through automation that results in lower staff costs.
-
Reducing batch processing.
A single business transaction often involves multiple applications working in conjunction with each other, and in this case your application integration environment must provide the logic that would otherwise be provided by people. This is achieved by defining business rules for the transaction to follow.
Figure 2.3 shows straight-through processing occurring with multiple applications.
Figure 2.3: Straight-through processing with multiple applications
Your decision on how extensively you use STP normally involves a tradeoff between initial costs and benefits. Dramatically reducing the amount of human interaction in your business processes can involve the creation of some very complex business logic and similarly complex IT solutions. You need to determine at what point the development costs exceed the benefits of extra automation.
Business Processing Capabilities Used in STP
Business processing capabilities are frequently used to provide straight-through processing. The only capability that is not generally used is the Workflow capability, because it normally applies to situations that require human interaction with the system. Table 2.5 summarizes the business processing capabilities used in STP.
| Capability | Usage |
|---|---|
| Rules Processing | Normally used |
| Business Transaction Management | Normally used |
| Workflow | Not used |
| Orchestration | Normally used |
| State Management | Normally used |
| Event Processing | Normally used |
| Schedule | Normally used |
| Contract Management | Normally used |
Data Capabilities Used in STP
Some data capabilities are essential for STP, but others are used only in specific situations. When the structure and format of data differs across the applications involved in STP, you need the Data Transformation and/or Mapping capabilities. If the flow is between different business domains, but within the same application working with the same data formats, you generally do not need these capabilities.
You may need the Data Access capability if the end-to-end response time of STP is unacceptable. By deploying the Data Access capability, you can ensure that data is staged for faster access, which should improve the overall performance of STP. If the overall response time is acceptable as is, you do not need the Data Access capability.
Table 2.6 summarizes the data capabilities used in STP.
| Capability | Usage |
|---|---|
| Data Transformation | Sometimes used |
| Data Validation | Normally used |
| Data Access | Sometimes used |
| Schema Definition | Normally used |
| Mapping | Sometimes used |
| Schema Recognition | Normally used |
Communications Capabilities Used in STP
Some communications capabilities are essential for STP. Others are used only in specific situations, and some should not be used at all. STP usually uses a messaging style of communication throughout the business process. This means that the messaging capabilities are generally required, but capabilities such as Request/Reply, Connection Management, Batching/Unbatching, and File Transfer are not. However, Request/Reply and Connection Management are useful for implementing some work-step invocation and response. File Transfer, while it is not the optimal solution, can be used as an alternative to messaging if your communications environment is based on file transfer.
The Message Forwarding capability enables indirect access to applications. This capability is normally not required in STP, which involves a concrete set of applications that are directly accessed. If you do need access to additional applications, you can make these applications part of the STP flow so that they do not require Message Forwarding.
Transactional Delivery is not always required for STP. Transactional Delivery is necessary in situations where multiple messages must be processed together or not processed at all through the constituent steps of a given STP flow. However, many STP flows do not necessarily involve such grouping of messages. Therefore, you should examine the specifics of your STP environment to determine if you need Transactional Delivery.
Serialization/Deserialization may be required if the structure of the data is very different across the applications that are part of the overall flow. If the flow is between different business domains within the same application working with the same data structures, Serialization/Deserialization is not necessary.
If the applications in the STP flow exist on different operating systems or represent data in different languages, you may require the Encode/Decode capability. Otherwise it is not normally needed.
Table 2.7 summarizes the communications capabilities used in STP.
| Capability | Usage |
|---|---|
| Message Routing | Normally used |
| Message Delivery | Normally used |
| Message Queuing | Normally used |
| Message Forwarding | Not used |
| Message Correlation | Normally used |
| Message Instrumentation | Normally used |
| Addressing | Normally used |
| Transactional Delivery | Sometimes used |
| File Transfer | Sometimes used |
| Serialization/Deserialization | Sometimes used |
| Request/Reply | Sometimes used |
| Batching/Unbatching | Not used |
| Encode/Decode | Sometimes used |
| Connection Management | Sometimes used |
Ensuring Data Integrity
In any application integration environment, data is generally passed from one location to another. In some cases, a master writable copy of the data exists, and other applications read from and write to that location. In such cases, your application integration environment needs to support only basic data replication. In other cases, however, multiple writable copies of the data exist. This data may exist in multiple databases, formats, and locations, and on multiple platforms. However, you still must maintain data integrity in each location.
To maintain data integrity in your environment, you should consider the amount and complexity of data that you are storing. You may have multiple writable copies of data in situations where a single master copy is more appropriate. Or maybe the format of your data can be standardized across your organization. Examining the data integrity requirements of your organization before building your application integration environment can save you a lot of money.
As an example of a data integrity problem, imagine that your company maintains employee information in multiple databases. The number of vacation days allowed for your employees exists in a human resources database, but salary information resides in a financial database. Both of these databases contain a record for each employee, but the records are separate. If you want to examine all of the records for an employee named Smith, you must retrieve records from both databases. If Smith marries and changes her name to Jones, you must modify records in each database to ensure that both are correct.
Data Synchronization
You can ensure that data integrity is maintained in your organization by using the data synchronization capabilities of your application integration environment.
Data synchronization is the act of reconciling the contents of two or more databases to bring them into an agreed-upon state of consistency, according to agreed-upon rules. Data synchronization is (at any one time) one-way, because data is changed in one location and is then copied out to other locations that require it. Depending on the requirements, the data may be copied when a particular event occurs, or at a particular time.
In a data synchronization environment, it is not generally practical for each program that changes one instance of common data to change all other instances. Instead, the data synchronization logic normally propagates the changes to the other copies at some later time. This behavior can lead to data concurrency issues. For more information about concurrency, see "Managing Latency" later in this chapter.
| Note | Data synchronization between two applications is often required in both directions. However, in these cases synchronization actually consists of two one-way processes rather than one two-way process. |
Figure 2.4 illustrates data synchronization between two applications.
Figure 2.4: Data synchronization between two applications
Business Processing Capabilities Used in Data Synchronization
The Rules Processing and Business Transaction Management capabilities are normally required for data synchronization. Rules Processing is required to validate the data. Business Transaction Management is required because data must be synchronized across all databases or none of them. All of the other business processing capabilities, however, are either sometimes used or not used at all.
The Workflow, Event Processing, and Schedule capabilities may be required, depending on the specifics of your environment. The Workflow capability is used when business users have to choose between duplicate instances of the same entity (for example, multiple addresses for the same person) during the synchronization process. Event Processing is required when synchronization depends on a given event (for example, the sale of a product registered on a point-of-sale device, which initiates a purchase order on the manufacturer's order entry system). The Schedule capability is used when synchronization must occur at a given time.
Table 2.8 summarizes the business processing capabilities used in data synchronization.
| Capability | Usage |
|---|---|
| Rules Processing | Normally used |
| Business Transaction Management | Normally used |
| Workflow | Sometimes used |
| Orchestration | Not used |
| State Management | Not used |
| Event Processing | Sometimes used |
| Schedule | Sometimes used |
| Contract Management | Not used |
Data Capabilities Used in Data Synchronization
As you would expect, data synchronization uses many of the data capabilities. The only capabilities that you may not need to implement are the Mapping and Data Transformation capabilities. Often, you do not need these capabilities, for example if data is synchronized across relational databases on a single platform that stores enterprise-level data in a consistent structure and format. You should examine the applications, platforms, and data involved in synchronization to determine the need for data transformation and mapping in your environment.
Table 2.9 summarizes the data capabilities used in data synchronization.
| Capability | Usage |
|---|---|
| Data Transformation | Sometimes used |
| Data Validation | Normally used |
| Data Access | Normally used |
| Schema Definition | Normally used |
| Mapping | Sometimes used |
| Schema Recognition | Normally used |
Communications Capabilities Used in Data Synchronization
Some communications capabilities are essential for data synchronization. Others are used only in specific situations, and some should not be used at all.
Data synchronization is, by definition, one-way. Even in situations where it appears to be bidirectional, you can think of it as two one-way processes. For this reason, Request/Reply, Connection Management, and Message Correlation normally are not used in data synchronization. Message Instrumentation is not used, because messages do not need to be processed in a predefined sequence. Message Forwarding is not used either, because data synchronization occurs directly with each database instance.
Other messaging capabilities are typically used as a means of communication for data synchronization. File Transfer may also be used for data synchronization. You will often decide whether to use File Transfer based on the capabilities of the applications involved and on the availability of other communications capabilities.
Serialization/Deserialization may be required if the structure of the data is very different in the databases that are being synchronized.
Synchronization usually applies to a given instance of a business entity. However, you may use the Batching/Unbatching capability to update a logical set of data records (for example, customer records and all the associated address records). Or you may use it to batch the records together for performance reasons.
You may require the Encode/Decode capability if the databases to be synchronized exist on different operating systems or represent data in different languages. Otherwise, it is not normally needed.
Table 2.10 summarizes the communications capabilities used in data synchronization.
| Capability | Usage |
|---|---|
| Message Routing | Normally used |
| Message Delivery | Normally used |
| Message Queuing | Normally used |
| Message Forwarding | Not used |
| Message Correlation | Not used |
| Message Instrumentation | Not used |
| Addressing | Normally used |
| Transactional Delivery | Normally used |
| File Transfer | Sometimes used |
| Serialization/Deserialization | Sometimes used |
| Request/Reply | Not used |
| Batching/Unbatching | Sometimes used |
| Encode/Decode | Sometimes used |
| Connection Management | Not used |
Managing Latency
If you have multiple sources of data, it will not be identical in all locations at all times. There will be a delay between the time that information is written in one location and the time that those changes are reflected in other locations. Your business requirements will determine what kind of time delay is acceptable.
As an example, imagine that your business provides services to your customers for a monthly fee. However, a customer has a complaint, and as a courtesy, you agree to reduce the cost of your services by 10 percent. This information is fed into your customer relationship management (CRM) database. However, in order for it to be reflected in the bill, the information must be synchronized with the billing database. In this case, the information must be in the billing database before the bills are issued, which could be in several days. However, if billing information is available immediately to the customer on the Web, you may need to ensure that the updates occur within minutes.
Concurrency Issues
Latency in the updating of data can cause significant problems, especially if the same record is updated in multiple locations. You must ensure that when these delays occur the resultant data is consistent and correct.
Consider the following approaches to maintaining data concurrency:
-
Pessimistic concurrency
-
Optimistic concurrency
-
Business actions and journals
The following paragraphs discuss each approach in turn.
Pessimistic Concurrency
Pessimistic concurrency assumes that if one instance of data is updated from multiple sources, the desired result will not occur; therefore, it prevents such attempts. In such cases, each time a user attempts to modify the database, the database is locked until the change is complete. Pessimistic concurrency can be problematic, because a single user, or client, may hold a lock for a significant period of time. It could involve locking resources such as database rows and files, or holding onto resources such as memory and connections. However, pessimistic concurrency may be appropriate where you have complete control over how long resources will be locked.
Optimistic Concurrency
Optimistic concurrency assumes that any particular update will produce the desired result, but provides information to verify the result. When an update request is submitted, the original data is sent along with the changed data. The original data is checked to see if it still matches the current data in its own store (normally the database). If it does, the update is executed; otherwise, the request is denied, producing an optimistic failure. To optimize the process, you can use a time stamp or an update counter in the data, in which case only the time stamp needs to be checked. Optimistic concurrency provides a good mechanism for updating master data that does not change very often, such as a customer's phone number. Optimistic concurrency allows everyone to read; and because updates are rare, the chance of running into an optimistic failure is minimal. Optimistic concurrency is not an efficient method to use in situations where the transactions are likely to fail often.
Business Actions and Journals
For situations in which data changes frequently, one of the most useful methods for achieving concurrency is to use business actions and journals. This approach relies on standard bookkeeping techniques. You simply record changes to the data in separate debit and credit fields.
For example, imagine that your company provides widgets for customers. The company receives two requests that alter the field containing the number of widgets. One request is a return of 20 widgets, and the other is a sale of 20 widgets. If you use business actions and journals, both requests can be received, and the field does not need to be locked.
In fact, this approach does not require the locking of fields even for an order change. For example, suppose that the customer changes an order from 20 widgets to 11 widgets. This could be thought of as a single change, but it could also be thought of as two separate changes—first an order of 20 widgets, and then a return of 9 widgets.
This type of data concurrency approach may involve updating each record on the fly any time a change occurs, or it may involve posting the information into a journal, which is checked for consistency and completeness before all of the postings are executed.
Business actions and journals do not prevent optimistic failures entirely. For example, if an update request uses previously retrieved information (for example, the price of a widget), the retrieved information may be outdated and may require validation within the update operation. Your design should specify what must happen when such an optimistic exception occurs, because you may not always want to notify the user of such an exception.
One other problem with business actions and journals is the potential for a modification to be processed twice. Two orders for 20 widgets is obviously different from one order for 20 widgets. If business actions and journals are not used and the master data is modified directly, the widget inventory field can be changed from 200 widgets to 180 widgets multiple times, and it will make no difference. Therefore, if your application integration environment uses business actions and journals, it must also ensure that each action occurs once and only once.
Data Aggregation
Some data must be merged together before it can be sent to other applications. For example, data may need to be combined from multiple applications, or from a single application over time.
The precise challenges of data aggregation depend on the number of records you are aggregating. Aggregation may take place one record at a time, or you may batch multiple records together to create one output message. Scaling up further, some batch integration solutions require the aggregation of thousands or even millions of records from dozens of applications. The processing requirements of data aggregation increase unpredictably as the number of input records grows to these levels. Consequently, a solution that works effectively for transactional data aggregation may not be able to keep up with batch data aggregation requirements. Therefore, in some cases you may need two separate data aggregation services—one optimized for smaller payloads and high transaction rates and a second service optimized for the batch processing of very large data sets.
Data aggregation can be particularly complex when different types of data arrive at the transformation service at different and unpredictable rates. An example is the merging of video and audio input, which requires synchronization over time. The solution often involves the use of temporary storage for data that must be synchronized with data that has not arrived yet.
Combining Application Functionality
Many IT applications require user input from presentation devices. The interactions are designed to suit human needs and often to support conversations with customers. However, as you move to a more service-oriented architecture, you will find that applications have different requirements, particularly to support semantically rich messages.
In most cases, it does not make economic sense to completely create your new applications from scratch. Often someone has already built some of the functionality of the application for you, and if you know that the functionality works, you can use it to build reliable applications cheaper and more quickly. You can build the new application by adding new programming logic to existing building blocks of functionality.
Application Composition
Application composition describes the functionality required to interpret messages from applications, make calls to other applications, extract relevant information from their replies, and pass the information back to the application that made the original request.
Composed applications have the following attributes:
-
Interaction between components is synchronous and two-way.
-
Interaction between components is fast, because a client application or user is waiting for a response.
-
As data is updated in one component, it is updated in other dependent components.
Figure 2.5 shows an example of a composed application.
Figure 2.5: A composed application
A composed application often provides the same functionality from a business perspective as STP. Which option you choose depends on how costly the implementation is in the short term, the medium term, and the long term.
Business Processing Capabilities Used in Application Composition
The Rules Processing capability is often used in application composition, because composed applications usually consist of reusable application components that contain business rules. Other capabilities may be used depending on the specifics of your environment.
The Business Transaction Management capability is often used to ensure that a complete transaction is successfully executed across all of the components in a composed application, but it is not needed in situations where a composed application does not have direct transactional responsibility (for example, in rules engines). The Workflow capability is required only in situations that involve human interaction. The State Management capability is used when application components have to resume processing after being in a suspended state.
Composed applications do not ordinarily rely on the Orchestration capability for multistep processing because orchestration logic is inherently embedded in the composed application. The Event Processing capability may also be built into composed applications; in fact, it is required for composed applications built out of silo applications, because it provides a mechanism for loosely coupled integration.
Composed applications do not use the Schedule capability because the communication is synchronous and two-way. Nor do they use the Contract Management capability. While contracts can exist between the application components that constitute a composed application, the management of those contracts is not related to the composed applications themselves.
Table 2.11 summarizes the capabilities needed to provide application composition.
| Capability | Usage |
|---|---|
| Rules Processing | Normally used |
| Business Transaction Management | Sometimes used |
| Workflow | Sometimes used |
| Orchestration | Sometimes used |
| State Management | Sometimes used |
| Event Processing | Sometimes used |
| Schedule | Not used |
| Contract Management | Not used |
Data Capabilities Used In Application Composition
In a composed application, there is no guarantee of the quality and state of the inputs received by the individual application components. Therefore, the composed application components must use the Data Validation and Schema Recognition capabilities to validate the inputs they receive.
Other capabilities may be required, depending on the specifics of your environment. Usually, composed applications are built from modular, reusable building blocks of functionality with generic inputs and outputs that conform to the same semantic definition. In such scenarios, the Data Mapping and Data Transformation capabilities are not required. However, if applications are composed across platforms, or if the components are extended into application domains for which they were not originally designed, Data Mapping and/or Data Transformation are likely to be needed.
Schema Definition is required at the entry point of a composed application, and in some cases the individual components need to use the Schema Definition capability. In other cases, the composed applications may be built across multiple reusable components that recognize a single schema. Finally, in situations where you require better end-to-end response times in your composed application, you may need the Data Access capability to stage data for faster access.
Table 2.12 summarizes the data capabilities used in application composition.
| Capability | Usage |
|---|---|
| Data Transformation | Sometimes used |
| Data Validation | Normally used |
| Data Access | Sometimes used |
| Schema Definition | Sometimes used |
| Mapping | Sometimes used |
| Schema Recognition | Normally used |
Communications Capabilities Used in Application Composition
Because composed applications use synchronous two-way communication, the capabilities used to support this communication are generally required, including Request/Reply and Connection Management. However, as mentioned earlier in this chapter, communication that is synchronous from the perspective of the application may actually be asynchronous at the communications level. In these situations, the Addressing capability is required to directly address the constituent components of the application, and the Message Delivery capability is used to provide a single casting delivery mechanism between the different components of the composed application. The Message Correlation capability is also required to match messages to replies.
Several capabilities are not used at all for application composition. Given the realtime nature of the interactions between the components of a composed application, File Transfer, Batching/Unbatching, and Message Queuing are not used. With no Message Queuing, there is no need for the Transactional Delivery capability. Message Instrumentation is not used, because the entry-point application that invokes the components of a composed application ensures that messages are processed in the proper sequence.
Other capabilities are used only in certain circumstances. For example, Message Routing is usually not required because composed applications ordinarily follow a predefined sequence of steps that use the functionality already available within the application. However, the capability is required in cases where the business needs to modify this sequence in a way that is not supported by the application.
In situations where components of a composed application represent multiple applications, you can use the Message Forwarding capability to ensure that each application receives the information it requires.
The Serialization/Deserialization capability may be required if the structure of the data is very different across the composed application components. Similarly, the Encode/Decode capability may be needed if the composed application consists of components that reside on different operating systems.
Table 2.13 summarizes the communications capabilities used in application composition.
| Capability | Usage |
|---|---|
| Message Routing | Sometimes used |
| Message Delivery | Normally used |
| Message Queuing | Not used |
| Message Forwarding | Sometimes used |
| Message Correlation | Normally used |
| Message Instrumentation | Not used |
| Addressing | Normally used |
| Transactional Delivery | Not used |
| File Transfer | Not used |
| Serialization/Deserialization | Sometimes used |
| Request/Reply | Normally used |
| Batching/Unbatching | Not used |
| Encode/Decode | Sometimes used |
| Connection Management | Normally used |
Managing Transactions
Managing transactions is challenging in any complex business environment because business processes do not consist of isolated activities. Instead, they consist of a series of related, interdependent activities. As you define your application integration environment, you need to include capabilities that help you manage transactions effectively.
Business applications are frequently required to coordinate multiple pieces of work as part of a single business transaction. For example, when you order goods online, your credit card is verified, you are billed, the goods are selected, boxed, and shipped, and inventory must be managed. All of these steps are interrelated, and if one step fails, all corresponding steps must be canceled as well.
Not only do you need to provide Business Transaction Management capabilities at the business process level, you also need to consider what happens at other levels of integration. Messages that pass between applications may also need to be passed as transactions to ensure that the system can roll back, in case one of a group of interrelated messages is not received properly.
Your application integration environment will normally include the following two capabilities to support transaction management:
-
Transactional Delivery. A communications capability that ensures that transactions satisfy Atomicity, Consistency, Isolation, and Durability (ACID) requirements.
-
Business Transaction Management. A business processing capability that ensures that when a transaction fails, the business can react appropriately, either by rolling back to the original state, or by issuing additional compensating transactions.
Providing ACID Transactions
Traditionally, transaction management has been thought of in terms of ensuring that transactions meet the Atomicity, Consistency, Isolation, and Durability (ACID) requirements, which are defined as follows:
-
Atomicity refers to the need to complete all the parts of the transaction or none of them. For example, when a pick list is generated to take a book from the shelf, the inventory quantity for that book should be decreased.
-
Consistency refers to the need to maintain internal consistency among related pieces of data. For example, in a funds transfer application in a bank, a $100 withdrawal from a customer's savings account should be completed successfully only if a corresponding $100 deposit is made in that customer's checking account.
-
Isolation refers to the need to ensure that the activities within one transaction are not mixed with the actions of parallel transactions. For example, if five simultaneous requests for a particular book arrive and the warehouse has only four copies of the book in stock, the five transactions must each run in isolation so that the orders are processed properly. Otherwise, it might be possible for each of the five business transactions to see an inventory level of five books and then try to reduce the number to four.
-
Durability refers to the need to ensure that the results of the transaction are changed only by another valid transaction. The results of the transaction should not be compromised by a power outage or system failure.
When an atomic transaction runs, either all of its steps must be completed successfully or none of them should be completed. In other words, when a transaction failure occurs, the underlying applications and data should be returned to the exact state they were in before the failed transaction started.
Developing an application that satisfies these ACID requirements can be extremely complex, and the resulting application can be hard to scale. To streamline application development and to ensure scalability, software vendors offer transaction management software that is designed to satisfy the ACID requirements. This software generally takes one of two forms:
-
Database management systems. Most enterprise-level database management systems provide transaction-processing features. Multiple applications can update multiple database tables or segments within a single, automatically managed transaction.
-
Transaction processing managers. When data has to be updated on several databases or file systems on several diverse hardware or software platforms, a transaction processing manager can coordinate all of these actions within a single business transaction. This coordination saves you the task of writing an application to manage and coordinate the updates to each of these data sources.
Transaction processing managers generally conform to a set of standards called XA standards, which are governed by the Open Group (http://www.opengroup.org). In an XAcompliant distributed transaction management environment, a single transactionprocessing manager coordinates the actions of multiple XA-compliant resource managers. A database, messaging system, file management system, or any runtime application environment can act as an XA-compliant resource manager as long it is able to perform the required transaction management tasks.
Transaction-processing managers can require significant processing power, high network bandwidth, and good management tools. Performance issues with transaction management are mostly associated with resource locking and communications overhead (for all of the extra communications traffic required to coordinate the highly dispersed transaction components).
Transactional Messaging
Transactional messaging is used when you want to perform several tasks so that all tasks, including non-Message Queuing operations, either all succeed or all fail. When you use transactional messaging, the sending or receiving application has the opportunity to commit to the transaction (when all the operations have succeeded), or to abort the transaction (when at least one of the operations failed) so all changes are rolled back.
In Message Queuing, a single transaction can encompass multiple sending and receiving operations. However, the same message cannot be sent and received within a single transaction. In a case where a transaction consists of sending multiple messages, either all of the messages are transmitted in the order in which they were sent (if the transaction is committed), or none of the messages are transmitted (if the transaction is aborted). If multiple messages are sent to the same queue, Message Queuing guarantees that they arrive in the destination queue exactly once and in the order in which they were sent within the transaction.
Messages that are to be delivered in the order in which they were sent belong to the same message stream. Each message in a stream carries a sequence number, identifying its order in the stream, and a previous number, which equals the sequence number of the previous message in the stream. The message whose sequence number equals the previous number of another message must be present in the destination queue before the message can be delivered. The first message in a stream always carries a previous number equal to zero.
A transaction can also consist of removing multiple messages from a queue. The messages are removed from the queue only if and when the transaction is committed; otherwise, they are returned to the queue and can be subsequently read during another transaction.
Transactional messaging is slower than nontransactional messaging because messages are written to disk more than once and greater computer resources are required. Transactional messages can be sent only to transactional queues on the local computer or a remote computer. When messages are received within a transaction, they can be retrieved only from a local transactional queue. However, messages can be retrieved in nontransactional operations from local and remote transactional queues.
Two-Phase Commit
Satisfying ACID requirements can be problematic if the components of the transaction are dispersed over some combination of multiple databases and transaction processing managers. ACID requirements are even harder to maintain when parts of the transaction are executed under the control of separate operational entities (for example, separate business units of a single enterprise or separate enterprises).
To satisfy ACID requirements in these environments, XA-compliant resource managers can control a two-phase commit. In a two-phase commit, the transaction processing manager first instructs each resource manager to prepare to commit. At that point, each resource manager writes the transaction's data to the database, file, or queue, but retains the data in memory. After all of the participating resource managers have successfully prepared to commit, the transaction processing manager issues a commit instruction. This is the second phase of the two-phase commit.
A two-phase commit is an extremely effective approach to satisfying the ACID requirements when different data stores and messaging systems must be coordinated. However, it is resource-intensive, with a lot of network traffic and a lot of interdependent system locks on pending local transactions. If an application or service is waiting for a reply and cannot continue until the reply is received, this approach can block business processes. A two-phase commit is appropriate for environments where each system is local to each other system; however, more distributed environments, particularly those using asynchronous transactions, require a different approach.
Transaction Compensation
When applications are loosely coupled through asynchronous transactions, a twophase commit usually becomes too problematic to implement because the atomicity of the transaction is difficult to maintain. In these situations, you can instead adopt an approach where compensating transactions are used to restore order when only part of a transaction is completed successfully.
As an example, consider a customer who is planning a vacation. When arranging the vacation, the customer must ensure availability of air travel, hotel, and rental car for the same week in the same location. If any one of these travel services cannot be arranged for that week in that place, then the entire transaction should be canceled.
For a tightly coupled transaction, the composite application has to contact the airline, hotel, and car rental reservation systems and keep each of them pending until all three can agree on a particular week. Because those activities may tie up transactions in the three reservation systems to an unacceptable degree, a better approach may be to handle this process as three individual transactions. In this case, if one or more actions cannot be completed, the other actions are canceled by executing a compensating transaction. Therefore, if the airline and car rental transactions are completed successfully, but the hotel reservation fails, then the composite application sends messages to the airline and car reservation systems to cancel those reservations. The composite application can then start over and look at availability for a different week or at a different resort.
The management of the logic required to recognize complete and incomplete distributed transactions and to issue compensating transactions is difficult to automate because each scenario can require different compensation strategies. Consequently, the process management capabilities of this integration architecture must readily enable developers to design and implement compensation logic that is matched to the business needs of each transaction.
With transaction compensation, the developer out of necessity takes an optimistic approach to transaction management. The assumption has to be made that the individual parts of the multisystem transaction will all be completed successfully. Many times that optimism is warranted. However, a loosely coupled transaction is not able to satisfy the ACID requirements of a distributed transaction as well as a transaction processing manager is. For example, isolation is problematic because multiple applications may each try to update the same data element within a very short time frame. Therefore, in the resort example, it is entirely possible for each of five travelers to believe that they have reserved the last rental car at the location that they are all visiting.
Also, if a transaction is cancelled before it's completed, it's not possible to have each system simply roll back those parts of the transaction that were successful. Instead, a compensating transaction must be run. For example, if the car is no longer available, a credit transaction has to be sent to the traveler's credit card company to compensate for the deposit that was just processed.
In addition to isolation, consistency is also compromised by transaction compensation. It's often easy to make changes to one system that are not immediately reflected in related systems.
| Note | Standards associated with transaction compensation include WS-Transactions, WSOrchestration, and BPEL4WS. For more information about these standards, see "Understanding Web Services Specifications," in the MSDN Web Services Developer Center (http://msdn.microsoft.com/webservices/understanding/specs/default.aspx). |
Long-Running Transactions
Some business processes, particularly those that involve human interaction, run over an indefinite time period. Traditional management techniques are not useful for these types of transactions, because each transaction holds database locks and resources. Because thousands of business processes can run on a computer at any one time, the number of resources held rapidly becomes impractical.
To get around the problem, these business processes are treated as long-running transactions. Long-running transactions are specifically designed to group collections of actions into atomic units of work that can exist across time, organizations, and applications. A long-running transaction usually contains several nested shortrunning transactions, the overall grouping (composition) of which is controlled by the long-running transaction.
In a long-running distributed business process, records in a database cannot be locked for extended periods of time, nor can records be locked in databases distributed across organizations. This enables other transactions to see the data being used by the transaction.
A long-running transaction satisfies all of the conventional ACID requirements except isolation. However, where appropriate, a long-running transaction may consist of a series of short-lived transactions which do ensure isolation.
As an example, consider a business process that is initiated when a purchase order request is received. The request is logged to a database and is then sent to the request approver. It might take several days for the approval response to be received, at which point the response is also logged to a database and the purchase order is sent to the supplier. Receiving and logging both the initial request and the response are each composed of multiple actions (receiving and logging).
In this scenario, you can use short-lived transactions to group related actions into a single atomic transaction (receiving a message and logging it to the database). However, you cannot group the receipt of the purchase request message and the receipt of the approval message within a single short-lived transaction, because that would lock rows in the database for indefinite periods. Instead, you use a longrunning transaction to group together the two short-lived transactions, which might be separated by a significant time period. Figure 2.6 illustrates this approach.
Figure 2.6: Grouping short-lived transactions into a long-running transaction
When the business process in Figure 2.6 is executed, the purchase request is received and the database is updated in a short-lived transaction. If anything goes wrong at this stage, the Receive PO transaction is aborted and no changes are made; otherwise, the transaction is committed. Then the schedule waits for the arrival of the approval message. When the message arrives, the database is again updated transactionally. However, if anything goes wrong at this stage, the To Supplier transaction will abort automatically. The Receive PO transaction cannot be aborted, though, because it has already been committed. In this event, the first transaction must supply a code that will perform transaction compensation.
Timed Transactions
Timed transactions trigger an abort of a long-running transaction if it has not been completed in a specified amount of time. It is often very difficult to decide in advance how long a business process should take. However, it is possible to make a reasonable estimate of how long a specific action within a business process should take (for example, the arrival of a message).
A timed transaction can group short-lived transactions and wait for the arrival of a message within a specified time period. If the message arrives in time, the timed transaction is committed; otherwise, the timed transaction aborts and causes the short-lived transactions to execute compensation code.
EAN: N/A
Pages: 50