Section 1.3. What forces created ESA?

1.3. What forces created ESA?

Modern businesses need functionality that is both distributed and centralized. Existing systems of record, such as Enterprise Resources Planning (ERP), Customer Relationship Management (CRM), Supply Chain Management (SCM), and so on, serve the needs of key segments of the organization. But at the same time, a need for many new processes has arisen that requires a flow that moves from one system of record to another, with the context for the process kept outside of any of the existing systems. The traditional way of building enterprise software is not well-suited to these new requirements and does not take full advantage of the new world of pervasive networks, reusable services, and distributed data. Treating an application as a self-contained world no longer meets the needs of business.

In the past, enterprise applications contained the end-to-end processes that were being automated. One program running on one computer automated a workflow process that began and ended inside that application. A single database was the central mechanism of integration. All elements of the stack were contained within one program, as shown in Figure 1-2.

Figure 1-2 actually shows a prettier picture than what exists in many mainframe applications. Even after workflow mechanisms were in use and points of integration were designated, process and integration logic ended up strewn all over the stack and was mixed in with application and UI logic. This structure, however, captures the spirit of mainframe applications, which at their best were organized into the following layers:

Figure 1-2. Mainframe and client/server architecture

• The UI layer

• The process logic layer (which controls the automation of the steps)

• The integration logic layer (which controls the way the programs interact)

• The application logic layer (which controls what the program is actually doing)

• The persistence layer that serves as the database (where all the information is stored)

From a development perspective, the mainframe and client/server tools gave developers control over a vertical slice of this stack, from the UI to the persistence layer. If functionality in other slices was to be reused, the developer would have a conversation with the developers of the other slices to figure out how to use their functionality. Everything came together and had to be carefully reconciled in the database, which was the central point of integration. One of the major points of ESA is to transform such conversations about reuse from an ad hoc event into a formal design based on the needs of business processes.

It is possible to access the functionality in a mainframe/client/server stack through application programming interfaces (APIs). But there was no template for this; each time an API was developed, it came with its own assumptions about how it worked and how it should be used.

The mainframe/client/server applications did anticipate the need for customization through metadata, different variables controlling an application's behavior, and templates for UIs. Customization, however, is a one-time reuse, and as we will see, services are created to be reused in a context unknown at design time. The mainframe/client/server stack worked well when used by competent hands, and huge leaps forward in automation and productivity were made based on this architecture. But because the stack was contained within one application, with the UI, process, application, integration, and information layers tightly coupled at design time, it was impossible to break open a mainframe application and restructure it to solve new problems.

1.3.1. Application proliferation: systems of record are built out

In the late 1980s and 1990s, ERP systems showed the power of the mainframe/client/server stack. Despite growing pains, the widespread success of ERPwith SAP leading the chargeled to the creation of other applications, as seen in Figure 1-3.

Figure 1-3. Many applications, many vendors

While ERP was focused primarily on only the financial and management aspects of a companybefore it expanded throughout the 1990s to sales, distribution, and other key functionsnew applications such as CRM, SCM, and supplier relationship management (SRM), among others expanded the range of automation.

This led to a proliferation of applications for most companies under the label of "best of breed." The idea was to get the best application for each purpose. This allowed the VP of sales to have the best CRM application, the VP of manufacturing the best SCM application, and so on. The main benefit of this proliferation was the creation of a comprehensive collection of systems of record that automated common business processes from end to end.

But solutions from different vendors created a problem because they took away the central point of integration in the mainframe/client/server world: the single database. Data was scattered all over the system landscape, or even worse, was duplicated in multiple systems. For example, the same customer data ended up in the CRM system and in the ERP system, sometimes with variations. Perhaps worse, it was difficult to get applications from different vendors to communicate properly, making it more difficult to maintain data consistency.

Communication and integration among applications became even more important when companies realized that essential processes may flow through several enterprise applications. The process that starts with taking an order and ends with the receipt of money, the so-called "order to cash" process, involved many enterprise applications. A financial transaction in the ERP system would move to the SCM system for a factory order, which then went to the CRM system for service questions, and then back to ERP for the final confirmation of the order. Other processes, such as procure to pay and sell from stock, had a similar cross-application structure and required better interaction among all the systems. The best-of-breed model left no one company in charge of making everything work, and whenever a company bought a new solution, it came with "some integration required." Getting it to work at all actually required expensive, hard-wired integration projects.

1.3.2. Bridging the gap among systems of record

The next challenge facing companies using enterprise applications was integration. How could all of the best-of-breed applications be made to work together to serve the needs of the cross-application processes that were becoming the key to increased efficiency and innovation? As shown in Figure 1-4, the key question concerned how to bridge the gap among systems of record.

Figure 1-4. Attempts at bridging the gap

Many different technologies emerged to bridge the gap, so a cross-application, integrated view of enterprise applications was created, based on the new possibilities of the Internet as a pervasive network and emerging technology standards such as HTTP, HTML, Java, and XML:

• Portals emerged as web-based UI technology that enabled one UI to connect to functionality from the different applications.

• Data warehouses collected data from all of the different databases within the applications in one place.

• Enterprise Application Integration (EAI) technology created engines that allowed one application to send an XML messagea standard for data formattingto another application. The receiving application could send a response back, and all sorts of fancy alerting, monitoring, and triggering could happen in central systems for routing and transforming messages.

• Business Process Management applications for process modeling and management were frequently coupled with EAI technologies to create a new way to define and execute processes in the center.

• Many of these integration tools were powered by application servers, a new sort of structure for applications based on standards such as Java 2 Enterprise Edition (J2EE) that were created for the world of the Internet.

These new technologies started to bridge the gap among isolated enterprise applications and enabled some cross-application coordination and development. The results were encouraging. Portals could bring together UI elements from different applications, as well as gathering information from different sources and displaying them in one place. Data warehouses created one view of distributed information, albeit with a delay caused by batch-oriented extraction processes. EAI technologies connected applications, but these connections were complex and threatened a new layer of unstandardized spaghetti. Parts of the gap were bridged with these approaches and the requirements for cross-application processes were met to some extent. These capabilities fell far short of a unified approach to UI, process, and information integration, however. They also ushered in a new set of problemsintegration of the integration technologies.

Portals might need to talk to the data warehouse, which may need to send and receive data through the EAI system, which could be working with a Business Process Management system. The same sort of integration problems that these technologies were designed to resolve among enterprise applications arose among the integration technologies, which also generally came from a variety of vendors. It was "best of breed" all over again, except this time, it concerned integration tools, not applications.

1.3.3. Consolidation: mySAP? Business Suite and SAP NetWeaver

The cost of integrating enterprise applications and integrating integration technologies quickly mounted, leading customers to ask, "Is this really our problem to solve?" SAP thought not, and solved this problem in two ways. First, SAP assembled its own solutions for ERP, CRM, SCM, SRM, and so forth into a unified collection called the mySAP Business Suite. Second, SAP integrated all of the integration technologies into a unified whole, called SAP NetWeaver. Furthermore, SAP started to develop all of its mySAP Business Suite applications using SAP NetWeaver: in other words, integrated applications built on integrated technologies as a platform.

This created the situation shown in Figure 1-5 in which an integrated set of tools could help manage processes across a set of enterprise applications designed to work together.

Figure 1-5. SAP NetWeaver and mySAP Business Suite

This approach solved a large portion of the problem of connecting enterprise applications and integration technologies to each other.

SAP NetWeaver is a single, integrated set of technologies used to unify a huge collection of integration and development functionality, including the portal, the data warehouse, EAI, application servers, Business Process Management, and a variety of other systems for supporting mobile devices, for constructing UIs, and for distributing data and managing master data.

SAP NetWeaver allows you to write programs not only in ABAP the language that has been used for more than 20 years to write applications in SAPbut also in Java. This helps solve the problem of integrating the integration tools, but still the problem of getting all of these applications to talk to each other remains.

Bringing all of the applications together in the mySAP Business Suite helped solve the second half of the cross-application integration problem in a variety of ways. Because the integration points between the enterprise applications started to become well-known, SAP could support them as part of the product, not as a special integration task. SAP was able to add business packages to configure enterprise applications to work together. This left the challenge of integrating legacy applications and products from other vendors into the mixand finally solving the "best-of-breed" dilemma.

The challenge still remained, however, to be able to recombine systems of record to solve new problems. The connections made possible by SAP NetWeaver allowed some processes to flow from one enterprise application to another, and solved a host of other problems as well. Much of the power of enterprise applications was still locked in the monolithic structure. Businesses needed to change faster than the connections between applications could be constructed.

1.3.4. The web services era

A new phase in the evolution of enterprise application architecture started with the emergence of web services based on XML. Web services comprise a family of interrelated standards that work together to provide a simple way to allow program functionality in different languages and on different platforms to interoperate.

Web services are based on sending XML back and forth in a way somewhat similar to EAI systems. But they use a simpler protocoltypically based on HTTPand inside it they can embed XML using protocols such as SOAP so that one web service can send another web service a message very easily. This interaction provides interoperability across applications and technologies from different vendors. Because every vendor supports the basic web services standards, messages can be passed from one service to another, regardless of the architecture of the underlying application.

Web services (described in detail in Chapter 14) emerged as a standard and became popular very quickly. Almost every vendor adopted and implemented basic web services standards immediately and, for once, a standard was quickly agreed on by the entire software industry.

The arrival of web services offered an exciting way to solve the problem of how applications from different vendors could communicate based on a common standard, as shown in Figure 1-6.

One implementation detail here is that SOAP is just one of the communications protocols that could be used with web services. It is possible for other protocols to be used as well, but in practice, most web services use SOAP through HTTP to ensure interoperability. Also, each web service is described in something called a Web Services Description Language (WSDL) file. This file could be stored in a repository that was defined by the Universal Description, Discovery, and Integration (UDDI) standard, which allows the storage, search, and retrieval of WSDL files.

Web services provided a jolt of excitement to IT because they promised a solution to the challenge of connecting many enterprise applications in a standard way. They also offered a solution to breaking apart layers of the application trapped inside the monolith. Any enterprise application could use web services to present its functionality for reuse. Any platform, such as Microsoft Windows, could use web services from another platform, such as Unix, because they were based on HTTP and other standards that were supported on all platforms. Web services could be implemented in any programming language. When programmers wanted to use something that was encapsulated in a web service, they didn't have to learn a proprietary API. And UDDI promised a way for everybody to learn what web services were out there.

Figure 1-6. Web services

The biggest new idea to emerge from the concept of web services was called service-oriented architecture (SOA). The idea of SOA had its roots in many other ideas and approaches that had been circulating in the computer science world for a while. CORBA, DCOM, and EDI are similar in approach and intent to SOA. The notion is simple: think of components as reusable parts and reusable services. It is a powerful notion, one that discards the underpinnings of the mainframe and client/server architectures. SOA was a collection of data and functionality that is reusable across different programs. In many ways, SOA is an external, cross-application form of object-oriented programming. SOA created reusable collections of data and functionality, which are similar to objects.

So with SOA, instead of a monolithic, big-black-box, mainframe type of application, you could build a series of services that you could recombine each time you needed to solve a new problem. You would not have to constantly start from scratch. The functionality in the layers inside the monolith were now set free. No longer tightly coupled to each other, the functionalities could be put to new uses. With this new approach came new terminology. Systems of record now also became known as service providers. Applications that used services from systems of record became known as service consumers.

The Web services and SOA spawned a lot of work on standards and how to standardize and build applications. XML was a very important way of expressing those standards. Standards started to emerge for making web services more secure to create standards for global data types for reliable messaging. The UDDI standard was created to help people find services. Development of industry standards was also accelerated. This raised the question of what sort of application would be built on top of service providers, as shown in Figure 1-7.

Figure 1-7. Service-oriented architecture

SOA became the blueprint for new forms of applications that bridged the gap and automated new processes based on services from systems of record. These applications clearly were going to be different from the mainframe/client/server applications. Everything would not be contained within one application. Processes could start in one application and be handed off to another, coordinated by the center. Communication inside the composite applications was going to be far more standardized, based on a unified approach to UIs, processes, and managing information. Data would be distributed across many different repositories. These new applications were called composite applications, as shown in Figure 1-8.

For companies such as SAP, which is focused on providing business value, SOA was a great idea, but primarily a technical one. Composite applications are the way that SOA would be brought to life and put to work. To create new functionality, you could use web services from existing systems of record to provide the base for a new composite application that might have to add just a few new services to perform a task. This made profound business sense because many years and much money had been put into creating these applications, and they worked well, solving the problems they were intended to solve. (Remember, you also can think of legacy applications as applications that work well to solve stable problems.) But new problems were emerging that needed to be solved, and innovation was becoming more important, not just as a way to secure a leadership position in the market, but also as a way to survive. Composite applications promised a way to use services from a variety of different enterprise applications that were exposed through web services and then to create a new sort of application.

Figure 1-8. Composite applications

And because composite applications were reusing parts of other applications, you could separate the process logic for a process such as order to cash from the underlying systems that were implementing it. That made it easier to change and optimize the orchestrated process and helped reduce IT as a bottleneck.

Moreover, UIs could be separated from the application logic, which allows composite applications to have UIs tailored for different process automation roles. For example, one person might need to see the process from end to end, and another person might need to see one step of the process. The UI could simply show what each role needed. All of this ended up creating a new sort of architecture for a composite application, as shown in Figure 1-9.

This simple vision for composite applications is particularly popular for companies selling professional services, as well as selling tools to build web services. The underlying assumption is that the arrival of web services and the tools to build them mean that IT departments should immediately begin building their own services and using them to create composite applications under the SOA umbrella. After the bill for creating and maintaining custom web services increases, this approach will again lead to the question, "Is this really the IT department's problem to solve?"

SAP's answer is no, and a simple automotive analogy explains why. A Porsche Boxster has 70 percent of the same components as a Porsche 911. The remaining sections of the Boxster are simply new parts added to the 911, producing an entirely new car altogether. This is possible because a huge inventory of reusable parts exists, and they were constructed specifically to be reused; just the idea of having reusable parts is not enough.

Figure 1-9. Composite application architecture

The idea of composite applications is not enough either. A reusable inventory of web services will allow the creation of new applications from existing components. But for this to work, you don't just need the idea of reusable parts; you need reusable parts that were constructed in a way that promotes reuse.