| This chapter will examine the most common form of integration being done today: the practice of performing bulk data transfer and batch updating using Extract, Transform, and Load (ETL) techniques. We will discuss the business impact of the latency and reliability issues associated with this method of integration, and examine prescriptive steps for migrating toward real-time integration using an ESB in the context of a case study using pattern sketches. Chapter 2 presented statistical data showing that enterprises aren't as integrated as you might think. Another stark reality is that most organizations don't have a good snapshot of their critical data at any given point in time. Amazon.com is well known for being completely automated, with instant access to all aspects of information in their supply chain, but this is the extreme exception to the norm. Due to the latency of batch updates and rekeying information from "sneakernet" integration, and the margins of error associated with those common practices, there is always some latency between business events and the processing and recording of the associated data. In a supply chain, this can result in not knowing what's in inventory at any given time. The business impact from this method of integration can cause large amounts of business capital to be unnecessarily tied up. ETL integration generally consists of a complex maze of applications, scripts, manual processes, and FTP file transfers. The process of an "overnight" batch transfer usually involves an application exporting its data to a common neutral format, such as an intermediary database or a flat file with some kind of delimiter to separate the data. The source representation is likely in a different format than the target; for example, the sending application may write all the data in plain text, and the receiver may expect the data in XML. The source data is run through a process to be transformed into the target format; the intermediary file is then transferred to another application using FTP, and imported into the other application using a merge-and-purge process (Figure 9-1). Figure 9-1. Extract, Transform, and Load is the most popular form of "integration" 
|
