Post-Implementation Reviews

Post-Implementation Reviews

A post-implementation review should be conducted after every BI project, regardless of whether the BI application runs perfectly or has problems. It is imperative to learn from each project in order to improve the quality as well as the speed of the development process for future BI applications.

A post-implementation review session is also an excellent forum for IT managers and business executives to become comfortable with the dynamic development process and the release concept of BI applications. In addition, the review session is an ideal venue for sharing the lessons learned with other project teams as well as with other business managers in the organization. This goes a long way toward making the necessary culture shift more natural and acceptable.

Topics to be reviewed can include schedule, budget, satisfaction, scope, negotiation skills, staffing, skills and training, project planning and reporting, and development approach (methodology), as well as contractors, consultants , and vendors or any other general topic. Table 16.2 lists some suggested review questions for those topics.

Table 16.2. Suggested Post-Implementation Review Questions

Post-Implementation Review Topics

Schedule

• Did the project come in on time?

• If not, why not? Was the schedule realistic? What slowed us down?

• How can we prevent delays next time?

Budget

• Did the project come in within budget?

• If not, why not? Was the budget realistic?

• How can we prevent cost overruns next time?

Satisfaction

• Are we achieving the benefits we expected in terms of the return on investment (ROI)?

• Are the online analytical processing (OLAP) tool and other access and analysis tools satisfying the analytical business needs?

Scope

• Were scope changes requested during the project? Were scope changes made as a result of prototyping?

• Was the impact analyzed and measured? What was the impact? Could it have been avoided?

• What did we learn about scope changes and the existing change-control procedure?

Negotiation Skills

• Were all requested functions and data implemented? Did the scope have to be renegotiated?

• Did other project constraints have to be renegotiated (time, quality, resources, budget)?

• Was the renegotiating process painless, or did it create friction between the business people and IT staff?

• What needs to be done to improve the renegotiating process?

Staffing

• Did we lose any key people during the project?

• Why did they leave? What was the impact of their departure ?

• How can we avoid that type of loss in the future?

• Was the core team staffed properly? Were there too few or too many team members ?

• Were the roles and responsibilities assigned appropriately?

• Did the team members work well together? Was there friction? If so, what was the reason for the friction?

• How can we increase team spirit and team morale in the future?

Skills and Training

• Were the skills of the team members sufficient? Was "just enough and just in time" training provided or was "emergency training" required during the project?

• Was the provided training effective? What should be done differently next time?

Project Planning and Reporting

• Did the team report "actual time" truthfully? If not, why not?

• Were the activities estimated correctly? If not, do we know why they were overestimated or underestimated?

• Does our procedure for tracking time and reporting project status work? How can we improve it?

• What other lessons did we learn about project planning, tracking, and reporting?

Development Approach

• Did we select the appropriate steps, activities, and tasks from Business Intelligence Roadmap ? If not, why not?

• Were important tasks left out? Were unnecessary tasks included?

• Did we use the operational prototype approach for application development? Did it work? What were the benefits?

Contractors, Consultants, and Vendors

• Did we effectively use outside consultants or contractors?

• Did they transfer their knowledge to our staff?

• What lessons did we learn from negotiating with vendors?

• Did the vendors follow the rules or try to go around them? How can we control that situation in the future?

General

• Was communication effective?

• Were business people available when needed?

• What other lessons did we learn? What should be done differently next time?

Organizing a Post-Implementation Review

Consider the following items when organizing a project review.

• How to prepare for the review: The project manager has to take some time to prepare for the review by:

  - Examining the issues log to see which issues were effectively resolved and which were not

  - Assessing the change-control procedure for its effectiveness

  - Reviewing the project plan to determine whether all the appropriate tasks were included

  - Studying the estimated and actual task completion times on the project plan to determine which tasks were underestimated and which were overestimated

  - Noting any problems with the technology platform, such as problems with tools or their vendors, hardware, network, and so on

  - Reviewing the budget to see if the actual expenditures came close to the estimated ones

  - Assessing the effectiveness of the training sessions

  All of these items are potential topics for discussion at the review.

• When to schedule the review: It is advisable to wait for two months after going into production before holding a formal review of the BI application. This will give the project team time to iron out all the glitches that are common during the first few weeks after "going live." It will also give the project manager and the business sponsor time to:

  - Review the project charter, project plan, project reports , project activities, and budget

  - Collect information and metrics about the usage of the BI application, the BI target databases, and the meta data repository

  - Organize the meeting

• Where to hold the review: The review session should be held offsite. Pagers and cell phones should be used for emergencies only; they should not ring during the session. The room should be set up as a conference room supplied with:

  - Several flipcharts

  - An overhead or data projector

  - Markers and masking tape

  - Two laptops, one for the facilitator and one for the scribe

  - Coffee ”lots of strong coffee

• How long the review should last: A well-organized, thorough review usually lasts two full days, especially for the first release of a new BI application. However, if time is in short supply, or if the release was small in scope and effort with no significant hurdles, one full day could be scheduled with the option of a follow-up session within two weeks if necessary.

• Who should attend the review: All team members from the core team and the extended team should be invited to participate in the review. They must be prepared to contribute. That means they must review the agenda and prepare to discuss the topics listed on it. They must also review any documents sent to them ahead of time and be prepared to discuss them. In short, every project team member should be an active participant!

• What to discuss during the review: A preliminary agenda should be published about four weeks before the scheduled review session.

  - The preliminary agenda should list all topics, including introduction and wrap-up , with estimated time allocations for each topic.

  - The time estimates must take into account the complexity of the topic and the number of people participating.

  - Everyone who is invited should be given the opportunity to add to the agenda and submit any pertinent documents to be reviewed.

  - About two weeks before the review session, the final agenda and all documents should be sent to the attendees.

Post-Implementation Review Session Flow

Post-implementation reviews are very structured and follow a prescribed procedure by which the group must abide. Figure 16.2 illustrates the typical flow of a review session.

Figure 16.2. Post-Implementation Review Session Flow

Certain people conduct certain parts of the meeting (Figure 16.3), as described briefly below.

• The business sponsor should open the meeting and give an introduction before turning the meeting over to the project manager. At the end of the session, the business sponsor should close the meeting.

• The project manager should discuss the flow, the rules, and the expectations of the review, then turn the meeting over to a skilled facilitator.

• The facilitator should lead the group through the topics on the agenda. The facilitator's responsibilities include the following:

  - Asking the person who owns a topic on the agenda to introduce the topic

  - Soliciting comments and feedback from the other participants

  - Assuring that the meeting does not get bogged down on any given topic

  - Monitoring the allocated time for each topic and interrupting the discussion when the time limit has been reached, at which point the facilitator must temporarily turn the meeting over to the project manager for a decision (see below)

• The " scribe " is a person who was not involved with the BI project. The main purpose for having a third-party scribe is to have a knowledgeable but neutral note taker who:

  - Documents the highlights of all conversations and comments

  - Documents identified action items and to whom they were assigned

Figure 16.3. Conducting a Post-Implementation Review

If more time is required for a topic, the project manager and the business sponsor must decide whether to continue the discussion beyond its allocated time or to cut the topic short and discuss the remaining topics on the agenda. In either case, a second meeting has to be called to either finish the discussion on the interrupted topic or to cover the other topics that had to be dropped from the agenda during this meeting.

At the end of the review session, all action items are reviewed, and the person to whom an action item was assigned estimates a completion date or a reply date (the date on which an estimate will be provided for the effort to complete the action item). The group must decide who will get the task to follow up on the action items and whether another meeting is necessary (and if so, how soon).