Persistence

In addition to processing information in flight, application integration technology requires that information consumed from a source system and published to a target system be persisted native to the application integration technology.

Persistence can take many forms, including simple logging to track the progress of information flowing from application to application, or even the invocation of remote application services (e.g., Web services). Or it can be more sophisticated, such as the ability to track and maintain state for long-term, durable business transactions, or even to provide some decision support capabilities through message warehousing.

Logging

Logging is the physical storage of information and processing that takes place on that information in a physical file or database that is local to the application integration technology. Its purpose is threefold:

• Auditing

• Fail-safe

• Transaction tracking

Auditing refers to the technology's ability to keep track of all information and service invocations during a specific period of time, and report on both the information and processing at any time for any reason. Auditing is very important because we need to keep track of the movement of information in support of business transactions, transactions that may have to be examined in the future to determine discrepancies or, perhaps, to track problems down within the application integration solution.

Auditing logs record all information that flows in and out of the integration server, as well as any transformation, filtering, or validation operations. We also track how other external systems operate on the data, such as those connected to the application integration technology using distributed standards such as Web services. Remember, the core issue that auditing attempts to address is accountability of the application integration solution for normal business processing.

What is more, auditing may be used to determine the health of the application integration solution and provide the ability to solve any problems that are noted. For example, by using the auditing facilities of an integration server, it is possible to determine message traffic loads, message content variations, and the amount of information requiring transformation. Auditing also tracks information that changes, the state of information before the transformation, and its state following transformation.

Fail-safe is another issue to address when considering logging. This means the information processed by an application integration solution is not only interesting from a business perspective, but is also required to recover the system in the event of a network or server malfunction. It's the role of a fail-safe logging mechanism to record and track information flowing in and out of the system, persisting the information as it's processed. By persisting the information during processing, the fail-safe mechanism is able to pick up where it left off after hardware and software failures, perhaps rolling the application integration processing back so the source and target systems are not left in an unstable or unreliable state.

Similar to fail-safe logging, transaction tracking logs the functioning of the integration server in support of transactions, including two-phase commit as well as short- and long-term transactions. This also relates to transaction controls (presented later).

Message Warehousing

A bit different from general-purpose persistence as described previously, a message warehouse is also mandatory. A message warehouse is a database that, as an option, is able to store information that flows through the integration server. In general, integration servers provide this message persistence facility to meet several requirements: message mining and message archiving.

Message mining allows the extraction of business data to support decisions, creating the message warehouse as a quasi-data warehouse. For example, it is possible to use the message warehouse to determine the characteristics and amount of new customer information that is being processed through the integration server. All new sales orders for a given period of time can be displayed. Off-the-shelf data-mining and reporting tools work wonderfully for such applications.

Information that is stored in the message warehouse is almost always stored without modification. However, in a few cases, the information must flow through a data warehouse as it would an aggregation or transformation process. The data is combined and altered so that it will make sense to the average business user. In general, application integration architects and application programmers accomplish this through the rules-processing mechanism of the integration server (or they may employ an outside application to alter the message for storage).

Message warehousing can provide services such as message integrity because the warehouse itself provides a natural, persistent state for message traffic. This also provides state-full messaging, or the ability to maintain states between two or more systems even when using asynchronous messaging; messaging that, by definition, is a cohesive rather than coupled mechanism.