PHASE ONE: INTEGRATION

Phase one represents the early adoption of Web services through “skunk works,” Proof-Of-Concept (POC), and pilot projects. During this phase, organizations will leverage Web services to develop a standard integration framework within the security boundaries of their firewall. Basic integration Web services will be implemented primarily leveraging XML and SOAP, but WSDL may also be used for early collaboration initiatives with trusted partners. Some early adopters have already committed significant financial and human resources to Web services implementations, while many more cautious organizations are just beginning to examine their potential. We believe that Web services will be leveraged as a mainstream integration tool during 2003 and early 2004 as the benefits of using Web services for integration become more apparent and compelling.

Business Landscape

The current technologies and approaches being applied to application integration are failing to achieve their business objectives. To date, integration solutions have been too complex, too difficult, too expensive, and too rigid to meet the constant changes of today’s dynamic business environment. The majority of Fortune 500 IT organizations list system integration as their number one challenge, and systems integration failures can be traced to nearly every aborted industry exchange and value chain integration project. Below is a recent example of how challenging and costly systems integration initiatives can be:

“Nike reportedly spent $400 million to overhaul its supply chain infrastructure, installing ERP, CRM and SCM—the full complement of analyst-blessed integrated enterprise software. So what happened? In the third quarter of last year, the Beaverton, Oregon-based sneaker maker saw profits drop by $48 million, year over year, thanks in part to a major inventory glitch (it overproduced some shoe models and underproduced others). Nike blamed one piece of its integration puzzle—its demand and supply chain management software—for the mix-up. (“This is what I get for our $400 million?” CEO Phil Knight famously asked, referring to the total cost of the integration project.) And what CIO reading the Nike tale didn’t feel an uncomfortable mix of emotions: relief that he wasn’t responsible for such a public and pricey screw-up, and worry that his own integration project could fail in just as spectacular a fashion.” ^[1]

Such challenges are not uncommon when integrating large, complex enterprise applications. The ongoing challenges of systems integration are a significant burden and arguably a distraction from the more strategic tasks of improving operation efficiencies and tracking too-rapid changes in the marketplace. As illustrated in Figure 3.2, Forrester Research estimates the average 2001 spending on internal systems integration to be $3.5 million, breaking down to 27% on software, 30% on professional services, and 42% on internal integration staff. Beyond the integration of internal systems alone, Forrester estimates that total system integration costs, including B2B integration for collaboration, are at $6.3 million in 2001 and will grow to $6.4 million in 2003.^[2]

Figure 3.2: The cost of internal application integration.

The need to focus so much effort on integration mainly derives from the heterogeneous application architectures that have evolved within the vast majority of organizations over the past 10 years. Many organizations have implemented Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), and Supply Chain Management (SCM) systems. While they are valuable management tools, enterprise applications frequently become information silos in which key business information is locked. Today, these enterprise applications are typically large clientserver systems implemented by using proprietary software and providing proprietary data integration tools. Beyond the enterprise applications, organizations typically have an abundance of custom applications and smaller departmental or niche package applications, all of which need to be integrated to provide a coherent view of the business operations and to reduce the need for duplicate data entry (which itself is error-prone and often leads to confusing and conflicting information).

The challenge of integrating these disparate information silos has been met with a mixed bag of system and data integration approaches, including:

• Ad-Hoc Custom Integration —Many application integration projects are implemented using a custom development approach on an ad-hoc, as needed basis. This approach, and technology used to implement the interface, might be proprietary to the systems being integrated, might adhere to an enterprise architecture, or more likely will be based on the skills of the individuals implementing the interface.

• Data Warehouse and Data Marts —Data is often extracted from operational systems and stored in data warehouses and data marts in an effort to provide a consolidated and consistent view of operational metrics for reporting and analysis. Depending on sources from which data is gathered, a data warehouse might reflect a near realtime perspective, but it more often reflects a weekly or monthly snapshot.

• Enterprise Application Integration (EAI) —In an effort to integrate the silos created by enterprise applications, EAI capabilities have been implemented. EAI makes it possible for data entered into one enterprise system to be replicated across other core systems. For example, a customer setup in an order processing system will automatically be created in the call center systems as well as all other core systems. You can think of EAI tools as enterprise middleware used to tie key enterprise systems together into the semblance of a cohesive whole.

Regardless of the approach used, the results of these system integration efforts are often typified by:

• Significant Investment of Time and Money —Integration projects, either as a single custom interface or as a large enterprise data warehouse or EAI initiative, represent a large percentage of current IT budgets. This huge investment reduces the funds available for other projects that may provide longer-term strategic value to the organization.

• Poor Data Quality —Through a lack of data definition standards and master data sources, it is extremely difficult to consolidate data from multiple sources without highlighting data errors and inconsistencies. This often leads to the need for individuals to identify the source of these inconsistencies and further contributes to limited operational visibility.

• Limited Operational Visibility —Current approaches to system integration struggle to provide the real-time operational visibility that executives require to effectively run their business. Too often, when faced with a critical business decision, the majority of time is spent trying to gather the right information rather than analyzing the data. In this scenario, executives are often left with too little information and not enough time to make the right decisions.

• Lack of Flexibility —Current systems integration techniques often result in tightly coupled interfaces between systems. The impact of this tight coupling is often manifested as significantly reduced organizational flexibility. As we witnessed with the advent of e-Business requirements, the internally focused applications and supporting systems lacked the flexibility required to enable e-Business processes. Organizations struggling to integrate e-Commerce web sites with back-office fulfilment systems in many cases resorted to manual internal processes. It certainly was not uncommon for orders submitted via a Web site to be manually rekeyed into back-office order management systems. The e-Business imperative exposed a deeper, more elemental problem for many organizations: lack of flexibility in the business processes underlying the enterprise and e-Business applications.

The advent of Web services promises to provide the tools required to tackle these issues while providing an architectural approach that leverages current application portfolio investments. These issues will not be resolved overnight, but early POC and pilot projects are a first step in the right direction. For some time, early adopters have been leveraging XML to develop Web services. Initially, two types of services are being developed: virtual services and custom services.

• Virtual Web Services —Most organizations will find that they have already implemented core business functionality several times, and that this functionality is replicated across a variety of business systems in the organization. Given that much of the functionality targeted for implementation as Web services already exists, why reinvent the wheel? Using Web services standards, a company can encapsulate functionality embedded in existing systems and expose that functionality as Web services.

  The key advantage of this approach is that firms are leveraging existing assets and investments as well as jump-starting the creation of Web services registries. As they begin to encapsulate existing systems, they will be able to rapidly build out their Web services toolkit, and minimize the need to custom develop new services from scratch. When possible, and when it makes sense, look at existing systems for the creation of virtual Web services before custom developing a new Web service from scratch.

  The vast majority of enterprise application vendors are already working feverishly to expose functionality from their systems as virtual Web services. And when it makes sense to upgrade enterprise applications, firms will be able to add new virtual services made available from each enterprise application. One important consideration here is that it might be more cost- and time-efficient for organizations to create a few focused virtual services of their own, where the availability or complexity of an enterprise application upgrade is an issue.

• Custom Web Services —Pilot applications are being developed from scratch as custom Web services. Initially, these projects are using Web services to tackle nonmission-critical applications, supplementing the creation of a service portfolio being developed from existing applications.

These services might be implemented in a three-tier model, as illustrated in Figure 3.3. As shown, the familiar presentation tier and data storage tiers are maintained, but rather than the middle tier being composed of a single unit of functionality, it is now composed of a number of Web services. These might be new custom services (identified with a “C”) developed for a specific requirement, or virtual services (identified with a “V”) where functionality from an existing enterprise application or custom applications is exposed as a service. It is important to appreciate that, unlike previous models used to develop the functional tier, once a service has been implemented it is relatively easy to reuse for multiple applications. Figure 3.4 illustrates this principle for a simple order status service.

Figure 3.3: Implementation of virtual and custom services.

Figure 3.4: Re-use of functional tier from ERP system.

This figure illustrates an example in which a virtual order entry Web service has been created by encapsulating the order entry functionality already available from the ERP system using Web services-based standards (specifically, XML and SOAP). As illustrated, the new order entry service is reused by the online purchasing system, the call center system, and the shipping systems. In this scenario the order entry service could provide the ability to enter a new order, maintain an order, or even check the order status.

Since the order entry service now resides between the ERP system and the other systems that use the order entry Web service, it provides a level of abstraction from the underlying ERP system. If, in the future, it is necessary to update or replace the ERP system, the order entry Web service can be used to separate the Web self-service, call center, and shipping systems from the implementation detail of the new ERP system. In effect the systems using the order entry service are abstracted from the implementation details of the ERP systems.

These early projects are a first step in a much longer multistep process that will ultimately see the migration of internal systems from multiple proprietary application interfaces and architectures to Service Oriented Architectures (SOA) based on open standards. Chapter 7, “Architecting for Competitive Advantage,” takes a closer look at SOAs as well as the broader architectural considerations for SOAs and Web services.

Planning Considerations In determining why internal integration should be the starting point for Web services implementation, consider the following thoughts:

• Cost Avoidance —The implementation of an Enterprise Application Integration (EAI) system is often an expensive and time-consuming process, with EAI software licenses costing between $500 thousand to $1 million, and the implementation costs for a single EAI system often running in excess of $3 million.

  “Enterprise Application Integration (EAI) is a journey, not a destination. Higher business value can be realized as increasingly complex business processes are automated, standardized, reused, and shared. Yet EAI costs can also be substantial from a financial standpoint and in terms of the organizational disruptions that are often involved.” ^[3]

  Even though EAI software can handle many of the integration problems between enterprise systems, EAI solutions are typically built using proprietary technology. This means that the data that was freed from enterprise systems may now be locked in the proprietary scheme of the EAI application. Web services promise to enable the next wave of EAI solutions using XML as an open, standards-based communication layer, breaking the proprietary nature of EAI tool sets and allowing open standards to be used as the basis for application integration.

  The vast majority of enterprise application vendors have either already implemented Web services interfaces to their application suites, or are currently in the process of implementing these changes.

• Skills Acquisition —The deployment of Web services for internal pilot integration projects provides a low-risk training environment in which the IT organization can begin learning the concepts behind Web services and the real capabilities of Web service development tools. Skills acquisition during this phase should primarily focus on XML, SOAP, and WSDL.

• Evolving Standards— The standards that will be at the core of Web services collaboration are either toward the outer bounds of the evolving tier of standards (for example, service publishing and discovery using UDDI repositories) or are still within the emerging tier of standards (such as business process execution using BPEL4WS tools). These standards are still experiencing a significant degree of flux and as yet are not recommended for mission-critical uses. For further information, refer to Chapter 2, “Standards, Concepts, and Terminology.” This rapidly evolving landscape creates a moving target for the development of broad-based collaborative Web services.

• Robust Security —By using Web services for internal integration, all activity occurs behind the corporate firewall. This means that the same security that protects all other existing core systems remains intact, and security software and policies are unaffected. Web services security standards remain an open issue. While it is possible to operate Web services with trusted partners over a Virtual Private Network (VPN), which may be a consideration for a closed network of trading partners, this type of security will not support the use of services accessed from public directories over the Web.

Internal integration will be facilitated using XML and SOAP to unlock information stored in application silos, from ERP backbones to CRM systems. Freeing up this information will significantly reduce the cost of systems implementation, allowing organizations to maintain a consistent view of business operations and metrics across the enterprise in support of real-time executive information systems and portal solutions for employees, customers, and suppliers.

Plan of Action

In order for the integration phase to be successful, the following steps are recommended:

1. Schedule executive briefings and initiate discussions regarding the strategic value of Web services for your organization.

2. Develop a broad-based Web services education and training program covering the high-level concepts and terminology. Completion of this training should be mandatory for management and executive roles.

3. Complete an inventory of the application portfolio and determine when each software vendor plans to rollout Web service-enabled versions of their applications. This will help firms decide when and where it makes sense to consider creating virtual services for reuse versus waiting for software vendors to add Web services interfaces to their packages.

4. Investigate industry-specific data definition standards and ensure your firm consistently uses those that apply to your organization. Where industry data standards do not exist, develop internal standards that can be used consistently across your organization.

5. Identify areas where the availability of real-time business intelligence will provide strategic business value to your executives. For example, availability of real-time channel inventory and Point of Sales (POS) data for the largest customers, cross-referenced with your own manufacturing schedules, might make it possible to better balance manufacturing capacity with anticipated demand. With enough forward visibility into your own, customer, partner, and supplier data, it becomes far more possible to manage proactively rather than reactively. Implement Web service-enabled, real-time executive dashboards, and collaborate with partners, suppliers, and customers to pool operating data, improving visibility into business network inefficiencies, and create opportunities to reduce operating costs.

6. Pilot Web services in areas where EAI tools might have been considered. This will result in immediate cost avoidance as integration problems are solved with open, standards-based technology.

7. While implementing early pilot projects, ensure both business and IT personnel are tasked with the identification and prioritization of both tactical and strategic opportunities in which Web services can be further leveraged. This is the ideal time to start prioritization of your firm’s internal Web services build-out.

8. Begin identification of critical business processes and functions that can benefit from Web services as your firm moves into the collaboration phase.

9. Continue to monitor emerging standards to determine when standards such as business process execution (for example, BPEL4WS) transition from the emerging to the evolving tier. This will indicate that the collaboration phase will imminently become a mainstream focus for many organizations.

The experience gained during the integration phase of the Web services adoption model will lay the foundation for the collaboration phase, both from a business and technical perspective.

^[1]www.cio.com, CIO Magazine, August 15, 2002, “Return on Investment,” by Sari Kalin.

^[2]Forrester Research, December 2001, “Reducing Integration’s Cost,” by Laura Koetzle.

^[3]EAI Journal, January 2002, “Building A Business Case For EAI,” by Christy Bass and J. Michael Lee.