6.7 Integration Design Patterns

Here is a collection of B2B integration design patterns. Some of the integration pattern names and concepts are modified from Yee and Apte (2001, pp. 30 “41), Integrating Your e-Business Enterprise , and expanded to cover Web Services technologies.

In each of the integration design patterns, a scenario is presented where the design pattern may be applied, followed by a sequence diagram to depict how Web Services technology can be implemented. Each integration design pattern will start by defining the context (background or requirements), problem (problem statement), force (design factors to be considered or when to use the pattern), solution (proposed solution depicted in Use Cases and sequence diagrams), risk (design or implementation risks, and the risk mitigation), and any relevant examples.

6.7.1 Application-to-Application Pattern

Context

It is fairly common to have ad hoc requirements for a point-to-point interface with another trading partner. For instance, Figure 6-12 depicts a scenario where a corporate institution (Customer B) wants to automate the Foreign Exchange trade order process by sending an electronic interface to the broker (Securities Firm A). Customer B is a large corporate customer, and it prefers to use a mutually agreed interface format or perhaps a standard data exchange format. Trading partners may use a customized adapter to transform business data from existing back-end systems to exchange with the trading partner (either synchronous or asynchronous).

Figure 6-12. Application-to-Application Pattern

Problem

Different trading partners' applications have local and proprietary data formats. These applications do not share a common data format, and thus they require customized data transformation. If a trading partner (such as Customer B) needs to send transaction data to another trading partner (such as Securities Firm A), it either customizes the data mapping and transformation or sends in a preagreed format before exchanging data. If the number of application interfaces increase, the customization effort for the data transformation or customization would be very expensive and inefficient.

Force

Different trading partners have different data formats that require customized data transformation. It may not be viable to standardize a single data format that can be used to exchange data with all trading partners. Thus, the use of a flexible data format (such as XML) and a standards-based messaging (such as SOAP messaging) is very crucial to facilitating application-to-application integration.

Solution

In this sample scenario (in Figure 6-12), Customer B wants to send a Foreign Exchange trade order directly from its back-end Order Management system in an fpML message format. Its current Order Management system does not process fpML message types. A custom adapter is used to extract and transform trade orders in fpML (for example, using JAXP). It then exchanges the fpML messages with the trading partner. The trading partner, Firm A, has a legacy FX trading engine and securities accounting back-office systems and does not support fpML. However, Firm A uses a custom adapter to receive the fpML message from Customer B and passes it to its Message Broker (aka EDI translator) to transform the data contents to a proprietary format that its back-office systems can understand. fpML is a market message standard, which is initially agreed upon between the two trading partners.

Figure 6-13 elaborates the details of the data exchange using Web Services. Firm A extracts and renders the data contents from the back-end applications. The Message Broker (the data transformation engine) then transforms the data into XML using JAXP. It uses a hashing algorithm to generate the digital signature (DSIG), attaches to the SOAP envelope, and sends the SOAP message using JAXM. Upon receipt of the SOAP message, Customer B's Message Broker (the data transformation engine) will unmarshal the SOAP envelope and retrieve the message content. It will also need to use the public key pair to decode the message content and transform the XML using JAXP into a format that is understandable by the back-end applications. Once the message processing is done, Customer B's Message Broker will then prepare an acknowledgement message (SOAP reply message), encode and sign with a digital signature, and return to Firm A's Message Broker. Upon receipt of the acknowledgement message, Firm A's Message Broker will notify the back-end application that a message reply has been received.

Figure 6-13. Application-to-Application Pattern Sequence Diagram

This Web Services integration pattern makes use of JAXP for flexible data transformation and JAXM for heterogeneous platform connectivity. It addresses the issues of proprietary data format and connectivity for point-to-point application interfaces discussed earlier. The "adapter" becomes a standards-based implementation and can be highly reusable for other trading partners.

When to Use

This Application-to-Application integration pattern is suitable for point-to-point exchanges where tight integration is required. There are limited numbers of trading partners. The message format is relatively stable.

EAI Versus Web Services

With an EAI implementation, architects and developers are likely to use an EDI translator (Message Broker) together with customized adapters. With a Web Services implementation, they may use JAXP to transform data from the back-end systems and JAXM to exchange fpML messages between the trading partners.

Risk

Not all trading partners are technologically ready to customize the data transformation engine to support SOAP messaging. Many customers are using proprietary EAI products for data transformation. It requires a long- term strategy to migrate all interfaces to support SOAP messaging.

Example

Many investment banks are taking online trade orders from large corporations or investment managers using a customized data transformation approach. For instance, RandomWalk's retail brokerage system, using its WOLF framework ( http://www.randomwalk.com/ ) is able to send a SOAP message to the trading partner. The SOAP message is used to encapsulate the trade order status or account balance. At the SunGard user conference 2001, SunGard ( http://www.sungard.com/ ) also announced that its system is able to exchange its trade information with OM products ( http://www.om.com/ ) using SOAP messages. These are examples of application-to-application data exchange. This pattern will be useful to streamline the order management process to support Straight-through Processing.

SOAP messaging using JAXM is a way to encapsulate business data in a SOAP envelope and send message acknowledgements to the trading partner. The "Paper and Pencil" exercise at the end of this chapter illustrates how to send a Foreign Exchange option using JAXM.

6.7.2 Standard Build Pattern

Context

One of the challenges of a massive global rollout of business applications is the complexity of software service versions to support the heterogeneous platforms used by different users or trading partners. For instance, in the banking industry, home banking and cash management systems are two typical examples in which different versions of user front-ends are deployed to the market. Supporting different software versions that run on heterogeneous platforms (such as Windows 95 and Windows 98) could be a nightmare.

Architects and developers may design and deploy a standard build gateway for both trading partners to exchange business data (either synchronous or asynchronous). All data formats and APIs are standardized on the same platform.

Problem

There are many variants (or versions) of application interfaces for different trading partners, even though these interfaces share many similarities. As the number of application interface variants increase, the maintenance and support effort will be immense and difficult to manage.

Force

Standardizing all data interfaces and gateways that handle data exchange will ease software deployment and management. However, many customers' business operating or deployment environments require localized requirements, thus deploying standard builds may not be easy or feasible in some cases.

Solution

Figure 6-14 depicts a scenario where both trading partners have agreed on a common interface with a standard API library build version 1.3. Securities Firm A has managed the deployment of a Standard Build Gateway product that can handle the data exchange using the standard API library build version 1.3. In this sample scenario, Customer B uses the APIs provided by the Standard Build Gateway, say version 1.3, to extract business data and transform the data in a standard message format. Then the Standard Build Gateway will exchange with the trading partner, who is also using the same Standard Build software with the same message format.

Figure 6-14. Standard Build Pattern

The Standard Build Gateway can be built in any arbitrary programming language and platform. Each of the back-end systems makes use of the common API library to extract data and send it to the Standard Build Gateway.

Figure 6-15 elaborates the details of the previous process of the standard build using Web Services. Firm A extracts and renders the data contents from the back-end applications. The Message Broker (the data transformation engine) then transforms the data into XML using standard build APIs. It uses a hashing algorithm to generate the digital signature (DSIG), attach it to the SOAP envelope, and send it the SOAP message using a customized SOAP-to-Standard-Build-Gateway protocol binding. Upon receipt of the SOAP message, Customer B's Message Broker (the data transformation engine) will unmarshal the SOAP envelope and retrieve the message content. It will also need to use the public key pair to decode the message content and transform the XML using standard build APIs into a format that is understandable by the back-end applications.

Figure 6-15. Standard Build Pattern Sequence Diagram

Once the message processing is done, Customer B's Message Broker will then prepare an acknowledgement message (SOAP reply message), encode and sign it with a digital signature, and return it to Firm A's Message Broker. Upon receipt of the acknowledgement message, Firm A's Message Broker will notify the back-end application that a message reply has been received.

The Standard Build Gateway may be a customized version of any existing communications gateway or a SOAP server with a customized set of APIs for data transformation, authentication, or exception handling. It is not necessary that a proprietary implementation be built from scratch. The customized SOAP-to-Standard-Build-Gateway protocol binding refers to the binding of SOAP messages to the underlying data transport of the Standard Build Gateway. If the Standard Build Gateway uses a JMS server, the SOAP-to-Standard-Build-Gateway protocol binding would simply denote SOAP messaging over JMS data transport. If the Standard Build Gateway uses a proprietary data transport implementation, then developers need to write their own data transport provider classes for the SOAP server to perform the SOAP-to-Standard-Build-Gateway protocol binding.

This Web Services integration pattern provides a standard build approach to address heterogeneous platform deployment issues. It is useful for the business scenario where the trading partners and their underlying applications are tightly coupled .

When to Use

This Standard Build Integration pattern is suitable for trading partners who have a strong desire to standardize all message formats and interface programs on the same platform. The Standard Build Gateway can also be deployed in a large number of sites. This allows tight integration between the trading partners and their underlying applications.

EAI Versus Web Services

With an EAI implementation, architects and developers are likely to develop some proprietary APIs and the Standard Build Gateway using the underlying EAI infrastructure. With a Web Services implementation, they may expose the common library as Web Services calls for the back-end systems to invoke. Similar to the Application-to-Application pattern, the Standard Build pattern uses JAXP to transform data from the back-end systems and uses JAXM to exchange fpML messages between the trading partners.

Risk

Although Standard Build deployment can standardize data format and interfaces, it also requires versioning management. The versioning of Web Services can be implemented by adding a version attribute in the XML message header and registering different software versions as different service bindings under the same business service in the Service Registry. Please refer to Chapter 4, Web Services Architecture and Best Practices, for more details.

Example

The Standard Build pattern has been widely used in highly centralized business environments such as the Treasury departments of international banks. For example, a top- tier British international bank deploys a Java-based Foreign Exchange system to its Treasury units worldwide. All data interfaces are provided by a Standard Build Gateway, which is deployed to the customer's site. Corporate customers (such as corresponding banks) can use their Foreign Exchange platform to connect to FXall exchange ( http://www.fxall.com ) via the Standard Build Gateway.

6.7.3 EAI Patterns

EAI Patterns are designed to provide some best practices that can be reusable for integrating with multiple systems within an enterprise. There are three variants of the EAI Patterns: Hub-Spoke Replication Pattern, Federated Replication Pattern, and Multi-Step Application Integration Pattern. They have many similarities in their approaches to resolving application integration issues. Very often, they implement the solution using a message-oriented middleware product.

Hub-Spoke Replication Pattern

Context

Large international firms or Application Service Providers often have numerous country offices around the world, each running an instance of the back-office systems. Typically, common reference data and business logic residing in these back-office systems need timely replication. The head office usually has all the customer and trade data as the master copy, and the local country offices act as spokes where the customer and trade data will be replicated. Figure 6-16 depicts a business scenario where common reference data and business transactions are replicated using a Hub-Spoke Replication Pattern.

Figure 6-16. Hub-Spoke Replication Pattern

Architects and developers can replicate business transactions to a remote trading partner or an affiliate's Web site (usually asynchronous). This assumes that the Service Provider hosts the business transactions centrally on behalf of its trading partners, affiliates , or remote country office, as in an Application Service Provider scenario. The replicated data is for local retrieval or convenience purposes.

Problem

Architects and developers need to extend the Business-to-Business integration to a larger number of trading partners (or remote country offices). Different trading partners or remote country offices run different systems on heterogeneous platforms. The Web Services deployment needs to be highly available to different users and to cater to the heterogeneous connectivity between the systems.

Force

Large-scale deployment of Web Services to a number of different trading partners (or remote country offices) requires a scalable Hub-Spoke Replication model. If all the customer and trade data reside in a relational database, then a database replication technology can be used to replicate and synchronize all hub-spoke instances. However, if the customer and trade data reside in different systems and none of them is sharing the same underlying storage platform, then Web Services technology would be a good solution to replicate and synchronize them from the hub to all the spoke instances.

Solution

In this sample scenario, Firm A receives a business transaction. It processes the transaction and stores it centrally in the master repository. It then publishes the transaction in messages to the remote spokes.

Figure 6-17 elaborates the details using Web Services to replicate the business and trade data from the hub to the spokes. A SOAP client (also acting as the transaction client) wants to update certain customer and account data to the hub. In this example, it is assumed that the SOAP client sends a SOAP service request with XQL calls to the hub and the hub replicates the business data with the spokes using SOAP messaging. The SOAP client sends a service request to update some account information to the hub using an XQL (or JDBC) call wrapped in a SOAP message. It establishes a messaging connection with the JMS queue manager and writes the business data to the predefined queues. It also uses a hashing algorithm to generate a digital signature (DSIG) with a pair of public and private keys and wraps the business data contents in a SOAP envelope using JAXM and/or JAXP. The business data (business transactions in SOAP messages) are then published to the hub using SOAP messaging over JMS. The publishing event will be logged for tracking purposes.

Figure 6-17. Hub-Spoke Replication Pattern Sequence Diagram

Upon receipt of the business data in SOAP messages, the hub unmarshals the SOAP envelope and extracts the data contents. It needs to verify the digital signature for authenticity of the sender's credentials using the public key. It will also notify the JMS queue manager that the hub is ready to update the business transaction. The hub transforms the data contents into its local repository or database. Now the customer and account data are updated in the hub, and the hub can start replicating the contents to the spokes.

The hub starts publishing the business data to the associated spokes. If the hub-spoke connectivity is done via a relational database, then the hub can initiate a database replication process. If the hub-spoke connectivity is done via an LDAP Directory Server, then the hub can initiate LDAP-based replication based on the current LDAP configuration settings. If the hub-spoke connectivity is heterogeneous (the spokes do not operate on the same platform), then the hub can send a SOAP message to all spokes that contains the new business data. Each spoke will then update its back-end database with the new business data. Upon completion, the spokes will return an acknowledgement SOAP message to the hub. The hub can also optionally send an alert notifying the SOAP client that the hub-spoke replication is completed.

This Web Services integration pattern provides a flexible replication solution to synchronize the hub and the spokes. It addresses the complexity of hub-spoke connectivity and the underlying platform interoperability. It is useful for central management of resources and business information.

When to Use

The Hub-Spoke Replication Pattern is suitable for highly centralized business applications under no geographical location constraints. The local spokes are set up for backup, faster access, or MIS for better performance.

EAI Versus Web Services

With an EAI implementation, architects and developers can use a multicast publish-subscribe model to simultaneously publish the transactions for all remote spokes to persist locally. Alternatively, they can use a database replication approach where remote spokes can pull from the centralized hub. With a Web Services implementation, they can use SOAP-JMS binding (for example, client initiating a SOAP message that contains the transaction, which is bound to a JMS client to publish or subscribe for the remote spokes or the ebXML Web Services technology (for example, Business Process Management).

Risk

Hub-Spoke Replication Pattern does not easily accommodate differences or variants in localized requirements.

Example

Many global organizations, such as computer manufacturing firms, have adopted this Hub-Spoke Replication Pattern to implement their global business operations.

UDDI Service Registry is a simplified example of the Hub-Spoke Replication Pattern. A SOAP client can update the business services and service details using JAXR. The hub (master UDDI Service Registry) can then be replicated to the spokes using the underlying platform technologies, such as DB2 database replication or LDAP replication. In this example, the SOAP client does not need to use complicated JDBC/XQL calls or JMS messaging because the JAXR is a high-level abstraction API that shields off the complexity of the underlying database or storage platform or the underlying hub-spoke replication mechanism.

Federated Replication Pattern

Context

Large international firms or Application Service Providers often have numerous country offices around the world, with each of them running similar back-end systems on different platforms. Typically, shared customer and account information in these back-office systems need to be synchronized in a timely manner to support CRM and cross-border trading. Figure 6-18 depicts a business scenario where shared customer and trade data are synchronized using a Federated Replication Pattern.

Figure 6-18. Federated Replication Pattern

Architects and developers replicate business transactions to peer hubs (usually asynchronous) hosted by the trading partner or affiliate's Web site. This assumes that the Service Providers and the trading partners are both hosting the business transactions.

Problem

Local market requirements cannot be easily accommodated by a highly centralized deployment strategy. Local offices and trading partners always have different requirements that need flexibility.

Force

Time to market and the need to meet local requirements are the key business drivers for the Federated Replication Pattern.

Solution

Figure 6-18 depicts a scenario where there are distributed hubs in different countries . The customer and account data need to be synchronized across the hubs. In this sample scenario, Firm A receives a business transaction. It processes the transaction and stores it centrally in its local repository. Then it publishes the transaction in messages to the peer hub for synchronization or backup.

Figure 6-19 elaborates the process with more details using Web Services to replicate the business and trade data between the hubs. The assumptions are similar to the scenario in Figure 6-17, except that the JMS queue manager will publish the business data to each hub in order to synchronize all hubs. A SOAP client (also acting as the transaction client) wants to update certain customer and account data to the federated hubs. In this example, it is assumed that the SOAP client sends a SOAP service request with XQL calls to each hub via the JMS queue manager and that the hub synchronizes the business data using SOAP messaging over JMS.

Figure 6-19. Federated Replication Pattern Sequence Diagram

The SOAP client sends a service request to update some account information to the first hub using an XQL (or JDBC) call wrapped in a SOAP message. It establishes a messaging connection with the JMS queue manager and writes the business data to the predefined queues. It also uses a hashing algorithm to generate a digital signature (DSIG) with a pair of public and private keys and wraps the business data contents in a SOAP envelope using JAXM and/or JAXP. The business data (business transactions in SOAP messages) are then published to the hub using SOAP messaging over JMS. The publishing event will be logged for tracking purposes.

Upon receipt of the business data in SOAP messages, the first hub unmarshals the SOAP envelope and extracts the data contents. It needs to validate the digital signature for authenticity of the sender's credentials, using the public key. It will also notify the JMS queue manager that the first hub is now updated. The first hub will then transform the data contents into its local repository or database.

Now the customer and account data are updated in the first hub, and the JMS queue manager can start publishing the business data to the other hubs. Each hub will perform similar hub processing. Upon completion of updating the business data content in the local hub, each hub will notify the JMS queue manager. When all hubs are synchronized, the SOAP client (the transaction client, or the client that initiates the transaction) will be notified.

This Web Services integration pattern provides a flexible replication solution to synchronize the federated hubs. It addresses the complexity of synchronizing different systems where customer and account data are updated in different times locally. It is useful for decentralized management of resources and business information, where a centralized hub may not be viable for various business reasons.

When to Use

This Federated Replication Pattern is suitable for highly distributed business applications. There may be geographical constraints for the distributed applications, and sometimes architects may prefer to partition different hubs for different transaction types. The federated hubs have autonomy of their own data. However, the originating hub owns the transaction while the other hubs are replicated with a copy.

EAI Versus Web Services

With an EAI implementation, architects and developers can use a multicast publish-subscribe model to simultaneously publish the transactions for all hubs to persist locally. Alternatively, they can use a database replication approach where remote hubs can pull from the originating hub. With a Web Services implementation, they can use SOAP-JMS binding (for example, a client initiates a SOAP message that contains the transaction, which is bound to a JMS client to publish or subscribe for the remote hubs) or ebXML Web Services technology (for example, Business Process Management).

Risk

A highly federated operating environment will result in many variants and software versions. This will make manageability of Web Services applications challenging. Versioning of Web Services will be particularly useful to a federated operating environment. Please refer to Chapter 4, Web Services Architecture and Best Practices, for more details.

Example

Many decentralized business environments (such as regional offices of investment banks in the United States, Europe, and Asia Pacific) implement Federated Replication Pattern, as they require highly flexible operations to accommodate local market requirements in a timely manner.

Multi-Step Application Integration Pattern

Context

For complicated application processing (such as trade financing or order fulfillment with logistics management), there are multiple steps or processes to be done in order to complete a business transaction. Each process may be performed by one application. These applications may also be sharing common business data and processing partial content at different stages of the life cycle. There may be dependencies between each application, so an Integration Manager would be useful to manage these processes and enable some common processes and customer data that can be reusable by these processes. Web Services technology can be used as the lynchpin to bind these processes together and share common customer data.

Figure 6-20 shows an example of the trade-financing scenario, where there are multiple steps to process a Purchase Order and a Letter of Credit. In step 1, Publisher 1 begins to publish a Purchase Order document (Message A) to the Information Bus (the messaging infrastructure). Subscriber 1 subscribes the trade data and sends to the Message Broker (step 2) to transform the content into a format (Message B) that can be understandable by the target recipient (for example, the seller who supplies the merchandise to the buyer). In step 3, Publisher 2 is responsible for sending the business document to the target recipient via the appropriate delivery channel. Subscriber 2 subscribes to the newly transformed data in Message C (in step 4). The process in step 5 will render and transform the business data into the email delivery channel.

Figure 6-20. Multi-Step Application Integration Pattern

Architects and developers implement a Multi-Step Application Integration scenario (usually asynchronous) where there are dependencies between processes, and applications may be distributed in different locations.

Problem

Some Web Services applications have dependencies, which are complex transaction workflows, and have multi-step routing rules for message processing. Similar processes (such as data transformation) need to be abstracted and made sharable to reduce duplicating efforts.

Force

There may be complex processing rules or routing rules for different types of XML messages. Some of the similar processing rules can be refactored. Web Services technology can be used to expose the common processing rules so that different applications can apply during the life cycle of the business transaction.

The Multi-Step Application Integration pattern is useful for complicated data transformation or workflow and for multichannel delivery support, which requires a chain of Web Services to be invoked.

Solution

Figure 6-21 elaborates the details of the processes to manage Multi-Step Application Integration using Web Services. The seller has delivered the goods to the buyer and is ready to collect his money from the sales with the Letter of Credit collection letter. A SOAP client (the seller) initiates a service request to apply for a Letter of Credit collection and invokes a trade finance collection Web Service that triggers a chain of business services. The trade finance collection Web Service will transform the collection letter into a financial EDI message and send to the remitting bank (the seller's bank) about the receipt of a collection letter. This will allow the remitting bank to validate the relevant trade and shipping documents, confirm release of the Letter of Credit documents, and enable the applicant (the seller) to collect the money via remittance.

Figure 6-21. Multi-Step Application Integration Pattern Sequence Diagram

In this example, the SOAP client submits a transaction message (for example, a Letter of Credit collection letter document) to the Service Provider. It transforms the transaction request into XML and publishes the transaction request to the Integration Manager. The Integration Manager manages multiple processes needed to handle the entire business transaction (in this case, processing the Letter of Credit collection). It will then publish a service request to the Message Broker via the information bus to transform the business data (Letter of Credit collection letter document) into an email format that can be sent to the trading partner (in this case, the corresponding bank that handles the Letter of Credit collection letter).

The Message Broker is a business application that transforms business documents from one format to another. It subscribes to any data transformation service request from the Information Bus. If there is a data transformation service request event from the Information Bus, the Information Bus will extract the business document and transform it into the format requested . Upon successful data transformation, the Message Broker will publish the transformed business document to the Integration Manager, who will relay it to the target delivery channel (in this case, it is an EDI message to notify the corresponding bank about the Letter of Credit collection letter).

This Web Services integration pattern eases the workflow management of complicated multi-step processes by using flexible messaging and data transformation. It also reuses existing similar processes and turns them into Web Service components . It is useful for complicated data transformation and for multichannel delivery support.

When to Use

This Multi-step Application Integration pattern is suitable for complicated data transformation with simple workflow requirements. There are also requirements for multichannel support to render the data contents to different formats simultaneously.

EAI Versus Web Services

With an EAI implementation, architects and developers can build custom adapters for the Message Broker component to subscribe to data transformation job orders and publish the transformed data contents back to the Information Bus. They will also need adapters for each of the different delivery channel components to subscribe to the transformed data contents. These delivery channel components include SMS gateways, WAP gateways, or EDI mailboxes. With a Web Services implementation, architects and developers can use SOAP-JMS integration similar to the previous EAI Integration Patterns.

Risk

Current Web Services workflow- related specifications (such as WSCI and BPEL4WS) are still maturing to handle complex multi-step application integration.

Example

Some B2B Exchanges (for example, Standard Chartered Bank's eXonomy) are implemented with this Multi-Step Application Integration pattern. eXonomy uses an ebXML manifest to encapsulate business documents (such as a Purchase Order or Letter of Credit) that can be passed to multiple applications within the enterprise. They have complex routing rules for processing a Purchase Order, for instance, and the order status needs to be delivered in multiple delivery channels such as email, a PDA, or a WAP phone.

6.7.4 Data Exchange Pattern

Context

Regional or international customers who open up their application platform to exchange trade data with other trading partners often require flexibility and scalability in handling multiple data formats and in interoperating with heterogeneous interface requirements from the trading partners. They also have to cater to high volumes of business transactions. This is especially complex and challenging if the customers want to provide white labeling services (aka act as an Application Service Provider for other customers), where they need to have a standards-based data exchange standard and an agile data connectivity (or exchange gateway).

Figure 6-22 shows a scenario where Securities Firm A has deployed a variety of order management systems (such as TIBMercury for Foreign Exchange trading) and a settlement system (such as SunGard STN). Customer B is a corporate customer of Firm A. Customer B places a Foreign Exchange trade order with Firm A. Firm A executes the trade order on behalf of Customer B. Upon successful execution, Customer B pays and settles with Firm A. At the end of the day, Firm A also needs to send an electronic file to reflect the daily trading activities, so that it can update its back-end account system (SAP/FI module).

Figure 6-22. Data Exchange Pattern

These are a series of data exchange activities between the two trading partners. Each data exchange requires a different data format. It would be ideal to use a common Exchange Gateway to handle all required data formats and various connectivity requirements. One common way to achieve the goal of interoperability is to encapsulate the business data in XML and wrap them in SOAP messages. In other words, architects and developers need to normalize all data from the back-end applications to XML and exchange the business transaction with the trading partner in XML (either synchronous or asynchronous).

Problem

Different trading partners and back-end systems require specific data format and database connectivity. The more trading partners a customer has, the more complex the data format and the connectivity will be. Thus, there is a need to simplify the development and management of the data format and connectivity between the trading partners.

A high volume of transactions using asynchronous Web Services needs to be handled during peak hours. Thus, the scalability and throughput of the data exchange is a key challenge.

Force

The Data Exchange Integration pattern is desirable for loosely coupled processes where multiple systems need to be integrated. It is good for a large number of trading partners, where system interfaces and service components can be reusable.

XML data format provides a flexible data format where proprietary data formats can be converted. Together with an XML parser, many systems can extract and transform their proprietary data into XML, which can facilitate data exchange with other trading partners.

Scalability and reliability of SOAP messaging using asynchronous Web Services is critical to the service level.

Solution

Figure 6-23 elaborates the details of business processes to support the Data Exchange pattern using Web Services. In this sample scenario, Customer B extracts and normalizes business transactions (for example, an FX trade order) in XML, transforms the business transactions into XML using JAXP, wraps the data content with a SOAP envelope, and then sends the data content in SOAP messages to Firm A's Exchange Gateway. Firm A's Exchange Gateway handles the network connectivity and the message exchange functions. It transforms the data content, using JAXP and XSLT, to a format that Customer B's applications can understand. Upon successful data transformation, Firm A's gateway will use a hashing algorithm to generate a digital signature (DSIG), attach it to the SOAP envelope, and send the data content in SOAP messages to Customer B's gateway using JAXM. When Customer B's gateway receives the SOAP messages, it will return with an acknowledgement.

Figure 6-23. Data Exchange Pattern Sequence Diagram

Customer B's gateway will verify the digital signature with a public key and use a hashing algorithm to decode the data content. Upon successful verification, it will send the data content to the back-end applications in SOAP messages using JAXM. Customer B's back-end applications will perform any necessary data transformation using JAXP. Then it will return acknowledgement to Customer B's gateway, which will also return acknowledgement to Firm A's gateway and Firm A's applications.

This Web Services integration pattern allows a flexible data transformation and connectivity for a large number of trading partners who operate on heterogeneous applications and platforms.

When to Use

This Data Exchange pattern is suitable for loosely coupled systems where there are large numbers of trading partners, and multiple applications need to be integrated simultaneously. It resembles the stock exchange-style architecture.

EAI Versus Web Services

With an EAI implementation, architects and developers can build custom adapters for each of the back-end applications to handle message transformation. As the trading partners may be using different network connectivity, architects and developers may wish to use different Exchange Gateways that handle different network connectivity, such as SMTP, EDI, or proprietary network connectivity (as in existing stock exchanges). With a Web Services implementation, the XML data transformation is usually implemented by JAXP and XSLT, and the Exchange Gateway is implemented using SOAP messaging and/or SOAP-JMS integration.

Risk

SOAP messaging without reliable and guaranteed message delivery is a key concern for delivering high-volume transactions in the Data Exchange pattern.

Example

Equities stock trading applications and stock exchanges are candidates for implementing this Data Exchange pattern. With the increasing interest in achieving Straight-through Processing in the capital market, many brokerage firms would like to adopt a standards-based data exchange mechanism, such as using XML to exchange trade order and settlement information. ISO 15022 XML is a classic example of trade data that are represented in XML and exchanged between trading partners using a private network (such as SWIFT) or the Internet. Web Services technology using the Data Exchange Integration pattern would be a good solution approach because it provides a standards-based data exchanging mechanism and does not require both ends of the trading partners to adopt any proprietary infrastructure.

There are increasing numbers of capital market vendors supporting SOAP or ebXML messaging for B2B integration and data exchange. Examples are TIBCO's ActiveEnterprise solutions for capital markets ( http://www.tibco.com/solutions/industry_solutions/finance/capmarkets.jsp ).

6.7.5 Process Integration Patterns

There are two variants of the Process Integration pattern: the Closed Process Integration pattern and the Open Process Integration pattern. Both have many similarities.

Closed Process Integration Pattern

Context

Business-to-Business integration often involves managing business workflow processes between trading partners. Some business processes do not need to depend on another. These are often called closed processes.

Figure 6-24 depicts a business scenario where each trading partner has a number of workflow processes to process the business transactions, prior to sending or receiving the business documents. Each trading partner uses a local workflow engine (Process Broker) to manage workflow and process integration. In this example, Customer B wants to get a Foreign Exchange rate quote from Securities Firm A. Upon receiving a rate quote from Firm A, Customer B calculates the risk in his portfolio management system and initiates a trade order. Firm A receives the trade order and calculates the risk based on Customer B's credit risk profile. Upon approval of Customer B's risk exposure, Firm A will execute the trade order. There is no dependency of Customer B's workflow events for Firm A's workflow events.

Figure 6-24. Closed Process Integration Pattern

Business process collaboration is often done by using both SOAP messaging and workflow-enabled processing tools. It also requires reliable messaging between trading partners. ebXML, using JAXM, provides more reliable messaging than SOAP messaging. It can help closed process workflow process because it has message provider classes that can provide message acknowledgement and resend messages later if the recipient is not available online.

Problem

There are complex workflow events within closed business processes. Simple SOAP messages may not be sufficient to handle workflow processing.

Force

Closed Process Integration refers to workflow processes that do not need to depend on one another. It is useful when there is tightly coupled integration between trading partners and there is a strong need for business service integration. Such Closed Process Integration is also ideal for a small number of trading partners creating some tightly coupled business services.

Solution

In this sample scenario, Customer B requests an FX quote from Firm A. Firm A responds with an FX quote. Upon Customer B's acceptance of the FX quote, Firm A will initiate a series of workflow events for account profile retrieval, risk exposure calculation, and credit checking. There is no dependency between two different sets of processes of each trading partner.

The Process Broker denotes intelligent business processes that can handle different workflow events and actions, as well as network connectivity and messaging with the corresponding Process Broker of the trading partner.

Figure 6-25 elaborates the details of the Closed Process Integration pattern using Web Services. Customer B publishes a workflow event (such as getting a Foreign Exchange rate quote) to his Process Broker. The Process Broker starts to process the workflow events and creates a service request using SOAP messaging. It uses a hashing algorithm to generate a digital signature, attaches to the SOAP envelope, and sends the SOAP messages to Firm A's Process Broker via JAXM.

Figure 6-25. Closed Process Integration Pattern Sequence Diagram

Firm A's Process Broker receives the service request. It verifies the digital signature with the public key and decodes the SOAP message using a hashing algorithm. It then publishes the associated workflow events (such as check risk exposure). Upon completing the series of workflow events at Firm A, the Process Broker returns an acknowledgement to Customer B's Process Broker and Customer B respectively.

ebXML messaging using JAXM is a good technology option to handle Closed Process Integration between two trading partners. Reliable messaging between trading partners is particularly important for business transactions containing high financial values. This Web Services pattern provides reliable messaging for business transactions, so that each trading partner can focus on managing its internal processes.

When to Use

This Closed Process Integration pattern is suitable for tightly coupled integration with a small number of trading partners and strong business service integration needs.

EAI Versus Web Services

With an EAI implementation, architects and developers can use customized workflow integration tools and preagreed message formats (or APIs) for exchange between the two trading partners. With a Web Services implementation, they can use ebXML Business Process Management tools.

Risk

Current Web Services workflow-related specifications (such as WSCI and BPEL4WS) are still maturing to handle complex multi-step application integration.

Example

Exception management for next -day settlement in the securities trading industry may be a good example. In order to manage better data quality for account information and trade transactions to support Straight-through Processing, there are many dependencies (such as account name validation and risk exposure) that need to be performed while settling a trade order transaction.

Sabre ( http://www.sabre.com ) is an example of this Closed Process Integration pattern. Sabre provides airline reservation services for a number of airlines. It uses an ebXML-based workflow engine to implement business process collaboration among its trading partners. Details can be referenced in Milo, Malks, and MacDonald Architecting and Delivering ebXML-based Collaborative Web Services (2003).

Open Process Integration Pattern

Context

Complex B2B integration often requires multiple business workflow processes with dependencies. Workflow processes that may depend on one another are called open processes. Sometimes, an external party, rather than either one of the two trading partners, handles workflow events (or shared public events), such as a call center or a managed Service Provider. In such a case (see Figure 6-26), the business documents that are exchanged between the trading partners need to reflect some information of the workflow event and the relevant workflow status. This would enable the intermediary (or the Process Broker) to route and process the workflow event accordingly .

Figure 6-26. Open Process Integration Pattern

Classical messaging approaches, such as EAI and SOAP messaging, are good for exchanging business documents. However, open process collaboration requires more sophisticated workflow processing and reliable messaging. Business process collaboration is often done by using both SOAP messaging and workflow-enabled processing tools. It also requires reliable messaging between trading partners. ebXML Business Process and BPEL4WS are examples of specifications to support workflow processing.

Problem

There are complex workflow events within open business processes. Open business processes can brand sophisticated business service offerings as one single service from one Service Provider, instead of silo processes from each of the Service Providers. This will depend on two key factors: reliable messaging to handle business transactions and business process collaboration that handles complicated workflow processes. Simple SOAP messages may not be sufficient to handle workflow processing.

Force

There are workflow processing rules and sequences for different Web Services applications. Trading partners may have workflow processes that may depend on one another.

Solution

The open process scenario has many similarities with the closed process scenario. In this sample scenario, Customer B registers customer profile with Firm A ("Register-me" event.) Customer B cannot initiate any FX transactions without completion of the registration ("Notify-me" status event).

The Process Broker denotes an intelligent business process that can handle different workflow events and actions, as well as network connectivity and messaging with the corresponding Process Broker of the trading partner.

Figure 6-27 elaborates the details of Open Process Integration pattern using Web Services. Customer B publishes a workflow event (such as get a Foreign Exchange rate quote) to his Process Broker. The Process Broker starts to process the workflow events and creates a service request using a SOAP message. It uses a hashing algorithm to generate a digital signature, attaches to the SOAP envelope, and sends the workflow events using ebXML Business Process (BPSS) or BPEL4WS to Firm A's Process Broker via JAXM.

Figure 6-27. Open Process Integration Pattern Sequence Diagram

Firm A's Process Broker receives the service request. It verifies the digital signature with the public key and decodes the SOAP message using a hashing algorithm. It then publishes the associated workflow events (such as check risk exposure). Upon completing the series of workflow events at Firm A, the Process Broker returns an acknowledgement to Customer B's Process Broker and Customer B, respectively.

This Web Services pattern supports the use of business process collaboration using ebXML BPSS or BPEL4WS. ebXML BPSS or SOAP messaging using JAXM can provide reliable messaging for processing business transactions. These technologies support business transactions with high financial values using reliable message services and address the issue of implementing complex workflow processes between trading partners.

When to Use

This Open Process Integration pattern is suitable for tightly coupled processes and technical integration with a small number of trading partners. The business scenario usually denotes a strongly "co-branded" service where there is some interdependency among the different sets of business processes of each trading partner.

EAI Versus Web Services

With an EAI implementation, architects and developers can use customized workflow integration tools and preagreed message formats (or APIs) for exchange between the two trading partners. With a Web Services implementation, they can use ebXML Business Process Management tools.

Risk

Current Web Services workflow-related specifications (such as WSCI and BPEL4WS) are still maturing to handle complex multi-step application integration.

Example

A private labeled online trading service (for example, a brokerage firm offering online trading services using its trading partner's existing online trading service and application infrastructure, but with its own branding) may be a good candidate. For instance, there are open business processes (such as customer registration and risk exposure checking) that depend on one another, as the private labeled Service Provider (in this example, the brokerage firm) owns the customer account information. The underlying application infrastructure depends on the provision of the customer account information from the private labeled Service Provider.

6.7.6 Broker Integration Patterns

There are two variants of the Broker Integration Patterns: Service Consolidation “Broker Integration pattern, and Reverse Auction “Broker Integration pattern. Both have many similarities.

Service Consolidation “Broker Integration Pattern

Context

There are increasing numbers of intermediaries (aka information brokers ) who offer portal services that consolidate or aggregate different business services. Content or Service Providers may be integrating with the portal service via URL rewriting, data exchange using XML and Web Services, or application-to-application interfaces (see Figure 6-28). Typically, a Partner Directory will be created (aka Yellow Pages) to list all business services. UDDI or ebXML Service Registry are usually used to implement the Partner Directory. Users (or subscribers) navigate in the Partner Directory and invoke relevant Web Services.

Figure 6-28. Service Consolidation-Broker Integration Pattern

Architects and developers consolidate business services from a number of service providers in order to provide an added-value service. Web Services technology can be used best to aggregate business information from different information sources or content providers. This enables users to have a single front-end to access multiple marketplaces . It would be a killer application to create user-stickiness.

Problem

Different sources of Web Services need to be consolidated to present a hybrid business service with added values.

Force

Different Service Providers have different integration and security requirements. Some Service Providers have lower service-level requirements (for example, their service is not around the clock), and some have very proprietary technology that cannot be easily integrated or scaled up.

Solution

In this sample scenario, customers search and discover a variety of FX trading services from the FX Brokerage Intermediary portal's Partner Directory. The FX Brokerage Intermediary aggregates information from different sources and consolidates it for customers for better customer service. There are a variety of integration points with different local technical integration constraints.

Figure 6-29 elaborates the details of Service Consolidation “Broker Integration pattern using Web Services. A Service Requester initiates a service request from the information broker (portal). The information broker performs a UDDI service lookup for the specified business service. The UDDI Service Registry returns the URI (service end-point) to the information broker. The information broker then invokes the business service. The remote business service provided by the Service Broker (Service Provider or content provider) requires authentication. The Service Requester will then submit credentials for authentication to the information broker and the Service Broker, respectively. Upon successful authentication, the information broker will send the service request in SOAP messages using JAXM. The Service Broker will acknowledge the SOAP messages and return with the business information as requested. The information broker consolidates the business information and returns the result to the Service Requester using JAXM.

Figure 6-29. Service Consolidation “Broker Integration Pattern Sequence Diagram

This Web Services pattern is useful for portal integration with multiple Service Providers or content providers. Traditional point-to-point partner integration requires complex technical customization for each partner's back-end application infrastructure. Using Web Services technology can lower the cost of partner integration because SOAP messaging is relatively flexible and less costly to implement. It can also accommodate different data formats and platform connectivity. Business information can easily be aggregated from multiple sources to create user-stickiness for the portal service.

When to Use

This Service Consolidation “Broker Integration pattern is suitable for a brokerage service with a single front-end. The Brokerage Intermediary is Service-Provider neutral. The integration approach is designed to cater for loosely coupled processes with a large number of trading partners.

EAI Versus Web Services

With an EAI implementation, architects and developers can use customized workflow integration tools to cater for different proprietary integration requirements and message formats (or APIs). With a Web Services implementation, they can use ebXML Business Process Management tools. Both XML document (asynchronous data exchange) and XML-RPC (synchronous data exchange) will be used.

Risk

The interoperability of different legacy technology used by the target service providers may be very challenging.

Example

Many emerging intermediaries, such as a third-party bill presentment service bureau , are examples of the Service Consolidation “Broker Integration pattern. Bumiputra Commerce Bank has deployed a portal service where SOAP messaging helps to extract the account information from different back-end information sources, then consolidates and delivers it to the bank's portal (refer to http://www.sun.com/finance/docs/bumiputra.pdf ). In the recent Worldwide Analyst Conference (February 2003), Sun demonstrated a Proof of Concept developed for a brokerage firm, where RPC-based Web Services technology is used to consolidate account balance and transaction details from various banks to create an aggregated financial portfolio (aka wealth management). These are good examples illustrating the Service Consolidation “Broker Integration pattern.

SOAP messaging with JAXM is a powerful messaging API to consolidate and aggregate different information sources to implement the Service Consolidation-Broker Integration Pattern. The "Paper and Pencil" exercise at the end of this chapter will provide practice on how to send an FX option using reliable ebXML Message Service via JAXM.

Reverse Auction “Broker Integration Pattern

Context

Architects and developers consolidate business services from a number of Service Providers in order to provide the lowest -priced service. This is a reverse auction-like service brokerage (refer to Figure 6-30).

Figure 6-30. Reverse Auction “Broker Integration Pattern

Problem

The reverse auction business model requires real-time messaging and integration capability to different Service Providers' back-end systems and the client's capability to respond to the offered price or services.

Force

Real-time messaging capability is required to support this auction-style business model.

Solution

In this sample scenario, customers do not need to search each Service Provider one by one from the FX Brokerage Intermediary portal's Partner Directory. They specify a criterium for the lowest FX transaction service from the FX Brokerage Intermediary. The FX Brokerage Intermediary aggregates information from different sources, then consolidates and filters information in order to locate the lowest-cost service. There are a variety of integration points with different local technical integration constraints.

Figure 6-31 elaborates the details of Reverse Auction “Broker Integration pattern using Web Services. A Service Requester initiates a service request from the information broker (portal). The information broker performs a UDDI service lookup for the specified business service. The UDDI Service Registry returns the URI (service end-point) to the information broker. The information broker then invokes the business service. The remote business service provided by the Service Broker (Service Provider or content provider) requires authentication from users. The Service Requester will then submit user credentials for authentication to the information broker and the Service Broker, respectively. Upon successful authentication, the information broker will send the service request in SOAP messages using JAXM. The Service Broker will acknowledge the SOAP service request and return with business information as requested. Finally, the information broker compares the prices to find the lower price and returns the result to the Service Requester using JAXM.

Figure 6-31. Reverse Auction “Broker Integration Pattern Sequence Diagram

This Web Services pattern is useful for reverse-auction style portal integration with multiple Service Providers or content providers. Traditional reverse auction requires highly customized point-to-point partner integration and real-time processing capability. Using Web Services technology can lower the cost of partner integration because SOAP messaging supports heterogeneous platform connectivity and real-time information processing (such as price comparison) at a lower implementation cost.

When to Use

This Reverse Auction “Broker Integration pattern is suitable for a reverse auction-like brokerage service with a single front-end. The Brokerage Intermediary is Service-Provider neutral. The integration approach is designed to cater for loosely coupled processes with a large number of trading partners. The services provided are price sensitive, time sensitive, and homogeneous.

EAI Versus Web Services

With an EAI implementation, architects and developers can use customized workflow integration tools to cater to different proprietary integration requirements and message formats (or APIs). With a Web Services implementation, they can use ebXML Business Process Management tools. Both XML document (asynchronous data exchange) and XML-RPC (synchronous data exchange) will be used.

Risk

Real-time and online Web Services can be very challenging to consolidate prices from this auction model.

Example

Auction-style Service Providers (such as eBay and Integral's CFOWeb) are potential candidates for this Web Services integration pattern.

6.7.7 How to Use Integration Patterns

In order to optimize the integration patterns, you may wish to define your integration requirements with Use Case modeling (for easier communication with vendors and partners) and chart your big picture with prioritized targets. This may not be easy as the more details you articulate , the more you see gaps and required changes.

Next, you may wish to look into the "When to Use" sections and map your integration requirements to each integration pattern to see which one is relevant Integration patterns are not mutually exclusive. Thus, you may need a hybrid or a mix-n-match of patterns.

Always use the integration framework as a guideline to identify components or areas of integration, and the protocols or standards to be used. Examine technology options for the technology used in the integration patterns. Once the integration architecture is ready and defined, you can begin to customize your integration methodology in details.

Table 6-4 and Table 6-5 recapitulate the discussion of the different integration scenarios and design patterns.

Table 6-4. Integration Patterns Summary 1

Integration Patterns	When to Use	Benefits	Consideration
Application to Application	Point-to-point exchange	Tight integration	Limited scalability
Standard Build	Strong branding Strong urge to standardize	Reduce deployment effort Standardized service, faster deployment with no customization	Consensus on standard builds
Hub-Spoke Replication Federated Replication Multi-step Application Integration	Hub-spoke business model Intra-enterprise integration	Flexible workflow integration Reliable and consistent multi-step application integration	Inter-enterprise integration with many customization options
Data Exchange	Large number of partners to integrate with heterogeneous platforms & standards	Accommodating differences in standards/interfaces	Emerging standards and technology
Closed Process Integration Open Process Integration	Shared business processes Workflow-oriented services	Richer support for process integration Cohesive and tightly integrated services	Complexity for partners to agree and implement
Service Consolidation “Broker Integration Reverse Auction “Broker Integration	Single front-end for multiple Service Providers	Added values and Service-Provider neutral	Handling service failure of partners

Table 6-5. Integration Patterns Summary 2

Integration Patterns	Typical Technology Used	Typical Standards Used	Examples
Application to Application	Customized adapters EDI translator	Proprietary XML variants	Ariba Commerce One
Standard Build	Proprietary	Proprietary	Hexagon
Hub-Spoke Replication Federated Replication Multi-step Application Integration	EAI solutions, such as Amtrix, Mercator, and TIBCO	JMS, SOAP-JMS binding	eXonomy
Data Exchange	XML Web Services	XML and SOAP, UDDI, WSDL	AIG Visa Commerce
Closed Process Integration Open Process Integration	EAI solutions or middleware, such as Sun ONE Integration Server EAI edition, XML Web Services technology	BPEL4WS
Service Consolidation “Broker Integration Reverse Auction “Broker Integration	Hybrid of any integration technology	Hybrid of any integration standards	Yahoo! Digilogistics (obsolete) CFOWeb Vcheq (obsolete) Bumiputra Commerce Bank