Lifecycle Iteration: Growing the DWBI System

Lifecycle Iteration: Growing the DW/BI System

Building a DW/BI system is an iterative process, so adding the next business process dimensional model is just like starting over, only easier. You still need to go through all the boxes on the Lifecycle, but you dont need to reinvent the wheel: you need to consider only the new stuff. For example, you already have your initial data architecture in the form of the bus matrix from your initial requirements gathering process described way back in Chapter 1. You also have the relative business value of each row in the bus matrix from the prioritization process also described in Chapter 1. (Remember the meeting where you worked with senior management to choose the highest value, easiest row on the bus matrix to implement first?) Now, go back and pick up the next business process on the prioritized list and move it through the Lifecycle. Its always a good idea to verify that business priorities havent changed with your business steering committee.

Business Requirements and Project Management

Business requirements definition is the cornerstone of the Lifecycle, but after the first round, it is usually limited to project-level requirements related to the new business process dimensional model. You will need more detail about this next business process: how do people use this information, what kind of analyses do they create or need, how does it tie in to existing data in the warehouse, what kinds of system or architectural implications do you see, and so on.

The initial part of every iteration will also involve some project planning, but this time it should be easier than the first time around. You now have real-world experience on which you can base your estimates. You already have most of your system infrastructure and development environment in place. You will be able to directly re-use several of the conformed dimensions and much of the ETL modules. In short, project planning should be significantly easier in the second and subsequent iterations.

After gathering project-level business requirements and creating project plans, youll continue through the Lifecycle by considering the technology, data, and application tracks.

The Technology Track

The second dataset often has significant technical and architectural implications you must deal with in the technology track. For example, the data in this iteration could have problems with data availability, definition, quality, and/or quantity. The technical issues may be something as simple as huge data volumes . Usually, these relate to the reasons you decided to leave this data until later. As a result, you may need to extend your architecture with additional functionality or capacity. In other words, you may need to buy some more software, or buy a new server, or both. For example, this second round may include a second source for customer data. In order to integrate the two sources, you may need additional software to help do the name and address matching. This second set of customer data may be much larger than the first set, driving a need for more disk space, and probably for more CPUs.

The great thing is, you already knew this from your requirements definition and prioritization. And, you set expectations early on that this would happen. In fact, you may have started the purchasing process long before you got to this point.

The Data Track

In the data track, you will develop and implement the dimensional model for this next business process. The modeling process is essentially a repeat of the in-depth dive you did in the first round to do the dimensional design. You will use all the existing conformed dimensions that apply, and typically add a few new dimensions that are specific to this next business process. For example, you may have implemented Orders data in the first round and now you want to bring in Shipments. Shipments will use the same Product and Customer dimensions, but it will also have a new dimension that describes the Shipper.

The ETL process is also essentially the same as in your first pass. Having the standards and examples in place can speed the process significantly, and not having to build all the dimensions saves time as well. However, the second and subsequent business process data models are often more difficult. They may have poor data quality, or large data volumes, or data integration issues, or any number of problems.

The Applications Track

In the Business Intelligence Applications track, you will add a set of standard analyses specifically related to the new business process. Often there are additional, and in many cases more powerful, analyses that incorporate data from both business processes now in the warehouse. For example, a retailer might have loaded sales data in the first iteration of the Lifecycle and is loading external customer demographics data in the second pass. As we described in the data mining chapter, the ability to combine behavioral data, in this case Sales, with descriptive data, in this case demographics , can be extremely powerful. You could use this combination of data to identify certain customer groups and offer them products they are more likely to be interested in. This is where the business intelligence system has a huge opportunity to add business value. Note that this has implications for the technology trackyou are not only adding new data, but you will be adding data mining functionality as well.

Deployment, Maintenance, and Growth

As in the first iteration, the three tracks come together in the deployment phase. You will need to train new users, and may want to update the training materials to reflect the new data. You will need to create documentation for the new business process data model and the associated BI applications and make it available on the web site. You will need to complete the full testing cycle before you release the data, and of course, you will need to support both the existing and new users.

Then, once this iteration is complete, you start all over again with the next row on the matrix. Remember that the work of the data warehouse is never done.