| Due to the fast flurry of vendors trying to gain attention in the growing ESB product category, combined with the number of industry analysts and journalists reporting with their opinions on it, understandably there is much confusion as to what an ESB actually is. This section serves to outline the main characteristics of an ESB. 1.7.1 Pervasiveness As illustrated in Chapter 1, the ESB can form the core of a pervasive grid. It is capable of spanning an extended enterprise and beyond, having a global reach across departmental organizations, business units, and trading partners. An ESB is also well suited for localized integration projects, and provides flexible underpinnings enabling it to adapt to any type of integration environment. Figure 1-2. ESB forms a pervasive grid that can span a global enterprise network 
Applications plug into the bus as needed, and are capable of having visibility and of sharing data with any other applications or services that are plugged into the bus. While web-services interfaces are an integral part of an ESB architecture, all applications do not have to be modified to become true web services to participate in the ESB. Connectivity is achieved through multiple protocols, client API technologies, legacy messaging environments, and third-party application adapters. 1.7.2 Standards-Based Integration Standards-based integration is a fundamental concept of an ESB. For connectivity, an ESB can utilize J2EE components such as the Java Message Service (JMS) for MOM connectivity, and J2EE Connector Architecture (JCA or J2CA) for connecting to application adapters. An ESB also integrates nicely with applications built with .NET, COM, C#, and C/C++. In addition, an ESB can easily integrate with anything that supports SOAP and web-services APIs, which includes de facto standard web-services toolkit implementations such as Apache Axis. For dealing with data manipulation, an ESB can use XML standards such as XSLT, XPath, and XQuery to provide data transformation, intelligent routing, and querying of "inflight" data as it flows through the bus. For dealing with SOA and business process routing, an ESB can use the Web Services Description Language (WSDL) to describe abstract service interfaces, and Business Process Execution Language for Web Services (BPEL4WS), WS-Choreography, or some other XML-based vocabulary such as ebXML BPSS, to describe abstract business processes. Don't worry if you don't know what all these buzzwords mean. Though this book is not intended to be an exhaustive reference or tutorial on all these individual technologies, they will be explained in sufficient detail in the context of how they relate to an ESB. These standards-based interfaces and components are put together in a meaningful way that comprises an open-ended pluggable architecture. An ESB provides an infrastructure that supports both industry-standard integration components as well as proprietary elements through the use of standardized interfaces. Chapter 1 shows a simplified view of an ESB that is integrating a J2EE application using JMS and JCA, a third-party packaged application using a JCA-compliant application adapter, a .NET application using a C# client, and two external applications using web services. Figure 1-3. An ESB integrating a variety of disparate technologies 
1.7.3 Highly Distributed Integration and Selective Deployment The ESB draws from traditional EAI broker functionality in that it provides integration services such as business process orchestration and routing of data, data transformation, and adapters to applications. However, integration brokers are usually highly centralized and monolithic in nature. The ESB provides these integration capabilities as individual services that can work together in a highly distributed fashion, and can be scaled independently from one another. In Chapter 6 you will learn more about the ESB "service container," a core concept of the ESB, which allows the selective deployment of integration services. 1.7.4 Distributed Data Transformation A key part of any integration strategy is the ability to readily convert data formats between applications. Many applications do not share the same format for describing similar data. Data transformation is inherently part of the bus in an ESB deployment. Transformation services that are specialized for the needs of individual applications plugged into the bus can be located anywhere and accessible from anywhere on the bus. Because the data transformation is such an integral part of the ESB itself, an ESB can be thought of as solving the impedance mismatch between applications. 1.7.5 Extensibility Through Layered Services An ESB gives you all the required core capabilities for virtually any integration project, and can be augmented with layered technology to handle more specific uses. For example, specialized capabilities such as Business Process Management (BPM) software can process workflow-related business processes, and collaboration servers can provide specialized services for managing business partners. Specialized third-party translators can provide data conversion from external data formats such as EDI into a target Enterprise Resource Planning (ERP) system or onto the general bus as an internal canonical XML representation. 1.7.6 Event-Driven SOA In an ESB-enabled, event-driven SOA, applications and services are treated as abstract service endpoints, which can readily respond to asynchronous events. The SOA provides an abstraction away from the details of the underlying connectivity and plumbing. The implementations of the services do not need to understand protocols. Services do not need to understand how messages are routed to other services. They simply receive a message from the ESB as an event, and process that message. The ESB gets the message to anywhere else it needs to go. In an ESB SOA, custom integration services may be created, extended, and reused as ESB functionality. Application endpoints, which are exposed as services, can be constructed together with specialized integration enablers to form composite services and process flows that can be recombined and reused for various purposes, with the goal of automating business functions in a real-time enterprise. Chapter 7 will discuss SOA in the ESB in more detail. 1.7.7 Process Flow An ESB's process flow capabilities range from simple sequences of finite steps to sophisticated business process orchestration with parallel process execution paths using conditional splits and joins. These can be controlled by simple message metadata or through the use of an orchestration language such as BPEL4WS. The process flow capabilities of the ESB make it possible to define business processes that are local to an individual department or business unit, and that can also coexist within a larger integration network. This is something a hub-and-spoke integration broker or a BPM tool can't do very well on its own. Chapter 7 will examine the details of a distributed processing capability that provides highly distributed business process orchestration without the need for a centralized processing or rules engine. Process flow in an ESB can also involve specialized integration services that perform intelligent routing of messages based on content. Because the process flow is built on top of the distributed SOA, it is also capable of spanning highly distributed deployment topologies (even spanning oceans at times) without the need to be painfully aware of the physical network boundaries or multiple protocol hops between applications and services on the bus (Chapter 1). Figure 1-4. Orchestration and process flow spanning highly distributed deployment topologies across physical and logical boundaries 
1.7.8 Security and Reliability The connections between nodes on the ESB are firewall-capable. The security between applications and the ESB, and even between the ESB nodes themselves, is capable of establishing and maintaining the most stringent authentication, credential-management, and access control. Reliability is achieved by having an enterprise-capable MOM at the core of the ESB. The MOM core provides asynchronous communications, reliable delivery of business data, and transactional integrity. As you will learn in Chapter 5, this is not your traditional MOM technology from a decade ago. Requirements have evolved and matured since then, and a MOM core of an ESB must be capable of meeting today's requirements. 1.7.9 Autonomous but Federated Environment Traditional hub-and-spoke EAI broker approaches tend to have organizational boundary problems, which are sometimes caused by the physical limitations of the EAI broker's incapability of easily spanning firewalls and network domains. More importantly, even if a hub-and-spoke architecture is capable of being stretched out across organizational boundaries, it still does not allow the local autonomy that individual business units need to operate semi-independently of one another. One of the biggest problems associated with extending the reach of integration beyond the departmental level is the issue of local autonomy versus centralized control. As part of the business culture in most large corporate environments, each department or business unit needs to operate independently of one another. However, they still rely on shared resources, and reporting and accounting information that funnels into a common business function. In such an environment, it is not reasonable to impose an integration strategy that requires all message traffic to flow through a centralized message broker sitting in headquarters. This is not simply a technical obstacle; it is a corporate culture issue as well. In an environment of loosely coupled business units, it does not make sense for things such as business process flow between localized applications, or security domains, to be managed by a single centralized corporate IT function. Loosely coupled business units within an organization need to operate independently of one other. Each should have its own IT function and not have to think in terms of routing all its message traffic, or delegating control of its business rules and security domains, through a centralized integration broker at one location or the other (Chapter 1). Figure 1-5. Separate business units lack the necessary autonomy if using a centralized hub-and-spoke integration broker 
Local business units and departments need to have control over their own local IT resources, such as the applications running at their site. The integration infrastructure should support deployment topologies to support that business model with practicality. The ESB provides this deployment model, allowing for local message traffic, integration components, and adapters to be locally installed, configured, secured, and managed, while still being able to plug together the local integration domains into a larger federated integration network with an integrated security model (Chapter 1). Figure 1-6. Autonomous and federated, an ESB allows organizations to cooperatively federate operations across organizational boundaries 
The distributed characteristics of the ESB are achieved by abstraction of endpoint definitions from the physical deployment details and underlying wire protocols, along with orchestration and routing of data between those endpoints. The federated characteristics are achieved by the ability of the ESB to segregate and selectively traverse application domains and security boundaries. 1.7.10 Remote Configuration and Management In some business models it doesn't make sense to have local IT staff at each remote location, although there is still a need for a loosely coupled, autonomous yet federated integration network. To illustrate this point, let's examine a simple case study for an ESB deployment in the retail industry. A video rental chain can have hundreds or thousands of remote locations that all contain the same set of applications, and all operate in the same fashion with regard to inventory management, accounting, and reporting (Chapter 1). Figure 1-7. A video retail chain with thousands of remote stores, all containing the same set of applications 
Using an ESB, an integration blueprint can be established to handle the local communication between the applications at a remote store. This includes the interface definitions to in-store applications, the routing of message traffic and message channel management, and security permissions. It may also include integration components such as an application adapter, protocol adapter, or data transformation. This integration blueprint, or template, can be deployed and customized at each site and act independently of all the other stores (Chapter 1). Figure 1-8. ESB configuration blueprint deployed at each remote location and remotely configured and managed 
This remote deployment blueprint capability is not unique to retail it has applications across other industries as well. The unified remote management of federated integration realms is a key contributor to the success of any ESB deployment in a highly distributed environment. Secure, Reliable Messaging Links In addition to sharing data between applications locally at each store, these remote stores need to share information with headquarters to do accounting and reporting, credit management, and tracking of employee data. The remote stores may also need to share information with each other. For example, a large video chain may want to offer a service whereby a consumer can rent a video from a store close to home, and return it to another store near the office. Therefore, stores in the same geographic area need to be able to share information about the rental in a near real-time fashion. Because of the latency and resiliency issues of satellite networks between the remote stores and headquarters, it's not feasible to just maintain a constant centralized access point for all rental information. That transient information about what you just rented two hours ago needs to be shared, or be accessible with an integrated data-sharing link between the remote stores. Because the link between the headquarters and the remote stores is achieved using reliable messaging, the interrupts in network service due to unreliable satellite links are compensated for by the messaging layer. Also note that it is possible for the stores to link among themselves using a secure and reliable messaging channel across an Internet connection. |
1.7.11 XML as the "Native" Datatype of the ESB XML is an ideal foundation for representing data as it flows between applications across the ESB. The data that is produced and consumed by a vast array of applications can exist in a variety of formats and packaging schemes. While it is certainly possible for the ESB to carry data using any form of packaging or enveloping scheme that you like, there are tremendous benefits to representing in-flight data as XML, including the ability to use specialized ESB services that combine data from different sources to create new views of data, and to enrich and retarget messages for advanced data sharing between applications. Chapter 4 will explore a fundamental benefit of XML the ability to insulate an organization from the need to synchronize upgrades between applications and will discuss the philosophy behind distributed XML transformation services in more detail. 1.7.12 Real-Time Throughput of Business Data An ESB eliminates latency problems by providing real-time throughput into in-flight data as it travels between applications across the ESB. Currently, one of the most popular integration methods is nightly batch processing. However, bulk batch-processing integration strategies, nightly or otherwise, are prone to high margins for error and can cause delays in information retrieval. The resulting high latency in getting up-to-date information can be costly. Chapter 9 discusses the details of this, and explores how an ESB can be used to refactor your organization from nightly batch processing toward real-time throughput of business data. 1.7.13 Operational Awareness Operational awareness refers to the ability of a business analyst to gain insight into the state and health of business operations. An infrastructure that allows the timely tracking and reporting of data as it flows across an organization in the form of business messages in a business process is an invaluable tool in helping to achieve operational awareness. A separate category of products known as Business Activity Monitoring (BAM) has emerged to address the many issues of operational awareness. One of the benefits of using XML as the native data format for the ESB is that messages are not treated as opaque chunks of data. If all data between applications and services is formatted as XML documents, underpinnings are provided by the ESB that allow you to layer advanced capabilities on top of the ESB to gain real-time insight into the business data that flows through your enterprise. These capabilities, whether they are inherently part of the ESB or are enabled as an extension of it, represent a natural part of the common infrastructure that includes the routing, process flow, and underlying plumbing, and that don't require a separate third-party BAM product be bolted onto the side. Auditing and tracking capabilities that are a base part of an ESB allow you to monitor and track the health of your business processes and the flow of messages throughout your SOA. Value-added services such as data caching, data collection and aggregation, and visual rendition of XML data can create a substrate for generating operational alerts, notifications, and reporting that can provide real-time insight into the state of your business as the data traverses your enterprise (Chapter 1). Figure 1-9. Value-added services enabling operational awareness provide real-time insight into live business data 
Tracking and reporting of data in the ESB is made possible by defining auditing and tracking points within a process flow, which in turn provide insertion points for collecting important content from business messages as they travel through a business process. Examples of trackable data are the business messages themselves, and the process events indicating whether a message has passed through a particular set of business process steps. Advanced value-added services can provide the data collection services, the query mechanisms, and the reporting capabilities that enable all this data to be gathered and presented in a meaningful way. XML persistence services can provide caching and aggregation points that can collect data to be transformed for the purpose of providing data to feed into other applications, or to feed into human-readable reporting mechanisms that can be used by business analysts. This means that data flowing through the ESB can be analyzed in real time to produce live information about the nature of your business for example, to provide a realistic snapshot of where your inventory is at all times within a supply chain. 1.7.14 Incremental Adoption One of the primary differentiating qualities of an ESB is its ability to allow incremental adoption, as opposed to being an all-or-nothing proposition. In the post-Y2K spending slowdown, budgets for multimillion-dollar IT projects are just not what they used to be. There is some indication that IT funding is getting freed up to solve the short-term tactical integration needs, but budgets are still being highly scrutinized at an executive level. At the same time, however, there is still a desire to implement larger corporate-wide strategic initiatives all of which rely heavily on integration and reuse of existing IT assets. Chapter 1 illustrates how an ESB can be used for small projects that each build into a much larger integration network. We'll see how this is accomplished as we delve into the details throughout this book. Figure 1-10. The ESB supports incremental integration, while working toward a more strategic goal 
The federated/autonomous capabilities of the ESB also contribute to the ability to adopt an ESB one project at a time. Incrementally staged deployments of ESB integration projects can provide immediate value while working toward the broader corporate initiatives. This notion of incremental adoption is also further supported by the ability to bridge into an existing integration broker hub and legacy message brokers. Integration broker hubs and their traits are explored in more detail in Chapter 2. | 