Message Warehousing

A message warehouse is a database that, as an option, is able to store information that flows through the integration server. In general, integration servers provide this message persistence facility to meet several requirements: message mining, message integrity, message archiving, and auditing.

Message mining allows the extraction of business data to support decisions, creating the message warehouse as a quasi-data warehouse. For example, it is possible to use the message warehouse to determine the characteristics and amount of new customer information that is being processed through the integration server. All new sales orders for a given period of time can be displayed. Off-the-shelf data-mining and reporting tools work wonderfully for such applications.

Information that is stored in the message warehouse is almost always stored without modification. However, in a few cases, the information must flow through a data warehouse as it would an aggregation or transformation process. The data is combined and altered so that it will make sense to the average business user. In general, application integration architects and application programmers accomplish this through the rules-processing mechanism of the integration server, or they may employ an outside application to alter the message for storage.

Message warehousing can provide services such as message integrity because the warehouse itself provides a natural, persistent state for message traffic. If the server goes down, the message warehouse may act as a persistent buffer, or queue, to store information that would otherwise be lost. Information may then be resent or compared with other message warehouses on the network to ensure message transfer integrity. The underlying principle is that of persistent message queuing supported by traditional message-oriented middleware. This also provides state-full messaging, or the ability to maintain states between two or more systems even when using asynchronous messaging, messaging that, by definition, is a cohesive rather than coupled mechanism.

Message archiving enables the integration server user to store months of message traffic in an archive for auditing or other purposes. It allows information to be restored for analysis. Many application integration administrators maintain message archives for just over a year, although no standard currently exists.

Auditing is the use of the message warehouse to determine the health of the application integration solution and to provide the ability to solve any problems that are noted. For example, by using the auditing facilities of an integration server, it is possible to determine message traffic loads, message content variations, and the amount of information requiring transformation. Auditing also tracks information that changes, its state before the transformation, and its state following transformation.