THE BUSINESS DIMENSION

only for RuBoard - do not distribute or recompile

First and foremost, this book is about data warehousing. Throughout the book, we will be exploring ways of designing data warehouses with a particular emphasis on the support of a customer relationship management (CRM) strategy. This chapter provides a general introduction to CRM and its major components . Before that, however, we'll take a short detour and review what has happened in the field of data warehousing, from a business perspective.

Although data warehousing has received a somewhat mixed reception , it really has captured the imagination of business people. In fact, it has become so popular in industry that it is cited as being the highest-priority postmillennium project of more than half of Information Technology (IT) executives. It has been estimated that, as far back as 1997 (Menefee, 1998), $15 billion was spent on data warehousing worldwide. Recent forecasts (Business Wire, August 31, 1998) expect the market to grow to around $113 billion by the year 2002. A study carried out by the Meta Group (Meyer and Cannon, 1998) found that 95 percent of the companies surveyed intended to build a data warehouse.

Data warehousing is being taken so seriously by the industry that the Transaction Processing Council (TPC), which has defined a set of benchmarks for general databases, introduced an additional benchmark specifically aimed at data warehousing applications known as TPC/D, followed up in 1999 by further benchmarks (TPC/H and TPC/R). As a further indication of the coming of age of data warehousing, a consortium has developed an adjunct to the TPC benchmark called The data warehouse challenge as a means of assisting prospective users in the selection of products.

The benefits of building a data warehouse can be significant. For instance, increasing knowledge within an organization about customers' trends and business can provide a significant return on the investment in the warehouse. There are many documented examples of huge increases in revenue and profits as a result of decisions taken based upon information extracted from data warehouses.

So if someone asked you the following question:

How many data warehouse projects ultimately are regarded as having failed?

How would you respond? Amazingly, research has shown that it's over 70 percent! This is quite staggering. Why is it happening and how will we know whether we are being successful? It's all about something we've never really been measured on in the past ”business benefit. Data warehouses are different in quite a few ways from other, let us say traditional, IT projects. In order to explain one of these differences, we have to delve back into the past a little.

Another charge that has historically been leveled at the IT industry is that the solution that is finally delivered to the customer, or users, is not the solution they were expecting. This problem was caused by the methods that were used by IT departments and system integration companies. Having identified that there was a problem to be solved , they would send in a team of systems analysts to analyze the current situation, interview the users and recommend a solution. This solution would then be built and tested by a system development team and delivered back to the users when it was finished. The system development lifecycle that was adopted consisted of a set of major steps:

Requirements gathering
Systems analysis
System design
Coding
System testing
Implementation

The problem with this approach was that each step had to be completed before the next could really begin. It has been called the waterfall approach to systems development, and its other problem was that the whole process was carried out ”out of the sight of the users who just continued with their day jobs until, one day, the systems team descended upon them their new, completed system. This process could have taken anything from six months to two years or more. So, when the systems team presents the new system to the users, what happens? The users say, Oh!, But this isn't what we need. And the systems project leader exclaims, But this is what you asked for! and it all goes a bit pear-shaped after that.

Looking back, the problems and issues are clear to see. But suffice it to say there were always lots of arguments. The users were concerned that their requirements had clearly not been understood or that they had been ignored. The systems team would get upset because they had worked hard, doing their level best to develop a quality system. Then there would be recriminations. The senior user, usually the one paying for the system, would criticize the IT manager. The systems analyst would be interrogated by the IT manager (a throat grip was sometimes involved in this) and eventually they would agree that there had been a communications misunderstanding. It is likely that the users had not fully explained their needs to the analyst. There are two main reasons for this. The first is that the analyst may have misinterpreted the needs of the user and designed a solution that was simply inappropriate. This happened a lot and was usually the fault of the analyst, whose job it was to ensure that the needs were clearly spelled out. The second reason is more subtle and is due to the fact that businesses change over time as a kind of natural evolution. Simple things like new products in the catalog or people changing roles can cause changes in the everyday business processes. Even if the analyst and the users had not misunderstood each other there is little hope that, after two years, the delivered system would match the needs of the people who were supposed to use it. Subsequently, the business processes would have to change again in order to accommodate the new system. After a little while, things would settle down, the teething troubles would be fixed and the system would hopefully provide several years of service.

Anyway, switched-on IT managers had to figure out a way of ensuring that the users would not have any reason to complain in the future and the problem was solved by the introduction of the now famous system specification or simply system spec. Depending on the organization, this document had many different names including system manual, design spec, design manual, system architecture, etc.

The purpose of the system spec was to establish a kind of contract between the IT department, or systems integrator, and the users. The system spec contained a full and detailed description of precisely what would be delivered. Each input screen, process, and output was drawn up and included in the document. Both protagonists signed off the document that reflected precisely what the IT department were to deliver and, theoretically at least, also reflected what the users expected to receive. So long as the IT department delivered to the system spec, they no longer could be accused of having ignored or misunderstood the requirements of the users.

So the system spec was a document that was invented by IT as a means of protecting themselves from impossibly ungrateful users. In this respect it was successful and it is still the cornerstone of most development methods. When data warehouses started to be developed, the developers began using their tried and trusted methodologies to help to build them. And why not? This approach of nailing down the requirements has proved to be successful in the past, at least as far as IT departments were concerned. The problem is that data warehouses are different. Until now, IT development practitioners have been working almost exclusively on streamlining and improving business processes and business functions. These are systems that usually have predefined inputs and, to some extent at least, predefined outputs. We know that is true because the system spec said so. The traditional methods we use are sometimes referred to as hard systems development methods. This means that they used to solve problems that are well defined.

But, and this is where the story really starts, the requirements for a data warehouse are never well defined! These are the softer We think there's a problem but we're not sure what it is type of issue. It's actually very difficult to design systems to solve this kind of problem and our familiar hard systems approach is clearly inappropriate. How can we write a system specification that nails down the problem and the solution when we can't even clearly define the problem? Unfortunately, most practitioners have not quite realized that herein lies the crux of the problem and, consequently, the users are often forced to state at least some requirements and sign the inevitable systems specification so that the solution can be developed. Then once the document is signed off, the development can begin as normal and the usual systems development life-cycle kicks in and away we go folks. The associated risk is that we'll finish up by contributing to the seventy percent failure statistic.

Is there a solution? Yes there is. All it takes is recognition of this soft systems issue and to develop an approach that is sympathetic with it. The original question that was posed near the beginning of this section was How will we know whether we are being successful? If the main reason for failure is that we didn't produce sufficient business benefit, then the answer is to focus on the business. That means what the business is trying to achieve and not the requirements of the data warehouse.

only for RuBoard - do not distribute or recompile