MILESTONE ANALYSIS

11.2 MILESTONE ANALYSIS

A status report provides the mechanism for regular monitoring of the project. It focuses primarily on whether the schedule is being met. A key advantage of status reports is that they do not require very much metrics data or analysis. This section discusses how metrics are used at Infosys to evaluate the state of a project at milestones.

If the project plan has been carefully derived and evaluated, the project will succeed if the plan is followed. Therefore, a milestone analysis should use the project plan and schedule as its baseline and compare the plan to the actual progress. This strategy is employed in the two common approaches for metrics-based tracking: the cost-schedule-milestone chart and the earned value method.⁵ Analyzing planned versus actual progress is considered a best practice for project management.³ At Infosys, planned versus actual is also tracked at milestones.

Because the main objective of metrics analysis for monitoring and control is to take corrective and preventive actions in a timely manner, such analysis should be done at regular intervals. For this reason, a project manager may define project milestones over and above the customer-driven milestones so that successive milestones are only a few weeks apart.

11.2.1 Actual Versus Estimated Analysis of Effort and Schedule

For schedule and effort, a project is likely to show deviation from planned in its actual progress. Small deviations on the effort or schedule front, however, can be considered "normal" and do not merit special attention. On the other hand, "significant" deviations may mean that the project may be heading for failure and hence call for further analysis and control actions.

To differentiate the normal from the significant, acceptable deviation limits are set in the project plan, as discussed in Chapter 7. Many projects at Infosys have a limit of about 20% for effort deviation and 10% for schedule deviation. The actual limits for a project are specified in its management plan.

If the deviation at a milestone exceeds these limits, it may imply that the project will run into trouble and might not meet its objectives; under time pressure, the project members might therefore start taking undesirable shortcuts. This situation demands that the project manager understand the reasons for the variation and apply corrective and preventive actions if necessary. Table 11.3 lists guidelines for analysis and possible control actions that a project manager may consider.

Table 11.3 includes suggestions for both types of variation: estimated effort too low or estimated effort too high. Some reasons are given, along with possible control actions. For example, if the estimate is too low, the possible reasons are that the estimate was too aggressive, resource utilization is low (that is, there is wastage in the project), a critical resource is not available, or team members have a low expertise level. For each reason, some control actions have been suggested. For example, if the estimates were too aggressive, the project manager might revise the estimates, request resources, or try to scale down the project. Most of the items in Table 11.3 are self-explanatory.

Table 11.3. Guidelines for Effort/Schedule Performance
Possible Reason	Actions to Consider
If Actual Is Less Than Estimate by More Than the Allowable Limit
Estimates for programs were too high or project team has more domain knowledge or experience than expected.	Reestimate for future modules.          Release resources.
The tasks so far have not been thoroughly performed.	Review tasks done so far and schedule reviews for work products not reviewed.          Examine issues log.
If Actual Is More Than Estimate by More Than the Allowable Limit
Low domain knowledge.	Schedule training.
Low software/coding experience of author.	Reassign to leverage existing experience.
New technology area.	Reestimate or request resources.          Negotiate delivery dates.
Estimates were too aggressive.	Identify main components for extra effort, and revise estimate for future activities.          Request resources.          Negotiate to scale down project objectives.
Resource optimization is low.	Reschedule and reprioritize tasks, and identify and eliminate "waiting times."
Nonavailability of a critical resource.	Escalate the issue.          Get a backup resource.          Reschedule, keeping critical resource(s) in mind.
Too much rework due to poor-quality of output of earlier phases.	Identify sources of problems and rectify them.          Change project schedule.

The pattern of effort deviation can also be useful for analysis. If the effort deviation has been consistently increasing from milestone to milestone, then even if it remains below the threshold, some action might be warranted. Similarly, if control actions are taken and the deviation percentage is reduced at the next milestone, it suggests that the actions are having the desired effect. On the other hand, if the effort deviation increases even more after the actions are taken, it implies that the previous actions are not working well and more drastic actions might be needed.

In addition to past performance, at a milestone the project manager may examine the prognosis for the rest of the project. For effort, in the milestone analysis, the number of resources available and needed is also reported. If the amount of effort available (based on the number of resources) is significantly lower than what is needed, clearly some action is required. Achieving this kind of visibility so that timely action can be taken is the purpose of this analysis.

11.2.2 Monitoring Quality

To monitor the third dimension quality the main metric is the number of defects found. In addition, the number and status of reviews done and the status and effect of defect prevention activities are also monitored. Because the number of defects was also estimated during planning, defects are assessed in the same way as schedule and effort. That is, using the defect levels predicted in the project plan for the various stages and the actual number of defects found, the project manager prepares an actual versus estimated analysis. If the deviation exceeds the threshold set for the project, she must analyze the causes for the deviation based on the decision she made on the actions to bring the project back under control. Table 11.4 gives guidelines relating to possible reasons and possible control actions. (Chapter 10 describes guidelines for evaluating reviews.)

For the quality control tasks of testing and reviews, in addition to the number of defects found, project managers also analyze the actual versus estimated for the effort spent in these tasks. These data are useful for analyzing the situation when the actual number of defects found differs substantially from the estimated. For example, if too few defects are found after system testing and if the amount of time spent in system testing is also too low, the reason for the testing performance becomes clearer.

Table 11.4. Guidelines If the Number of Defects Found Differs from the Estimate
Possible Reason	Actions to Consider
If Fewer than Estimate
Work product is of high quality.	Identify reason and see whether there are possible lessons for the project or for the process.
Inadequate testing.	Check the effort spent on testing; review the test plan and enhance it.          Schedule further testing.
Very thorough execution of earlier quality control activities.	Examine all review and testing records for the project.          Check whether there are possible lessons for the project or the process.
Defect estimates are too high.	Identify cause and correct estimates.
If More than Estimate
Inadequate execution of quality control activities so far.	Examine all testing and review records.          Schedule reviews of critical modules before continuing with testing.
Insufficient reviews and unit tests planned.	Enhance test plan and schedule further testing.          Review estimates and plans for acceptance.
Defect estimates are too low.	Identify cause and correct estimates.

The second metric monitored is the number of reviews conducted. As time pressures mount, there can be a tendency to skip the planned reviews. Because these reviews are an important part of the quality plan developed to achieve the project's quality goal, their satisfactory execution is essential. The milestone report therefore indicates which reviews were planned and which were actually executed. The actual performance of reviews is monitored at the completion of each review, as discussed in Chapter 10.

11.2.3 Risk-Related Monitoring

Risks and related activities are also monitored at milestones. As discussed in Chapter 7, risks for a project are not static; risk perceptions change over time and as risk mitigation steps are executed. Thus, it is important to take stock of risks and note the effects of the risk mitigation steps taken so far. For this reason, the current risk perception, along with the status of current risk mitigation steps, is reported in the milestone analysis report. Clearly, if risk exposure due to some risk has not been reduced, it implies that the risk mitigation steps are not having the desired effect. The risk and its mitigation steps must therefore be evaluated.

An important risk mitigation step is training, which is suggested to counter many types of risks. Training is also an important component in any project involving new technology or new people and if not done properly can introduce new risks. To prevent monitoring of training from slipping through the cracks, the milestone report describes the project-related training planned and actually executed.

Requirement changes also pose a risk to the project because they have an adverse impact on cost, schedule, and quality. Chapter 3 explains the process for managing requirement changes. In milestone analysis, a summary is reported of the changes that have been requested and their impact on effort and schedule.

Issues, if left unresolved for long, also frequently become risks. Hence, the status of issues is also reported in the milestone analysis. In addition, customer complaints, which clearly represent serious risks to the project, are also reported. In particular, the number of new customer complaints received, the number of customer complaints addressed, and the number of pending customer complaints are all reported.

11.2.4 Milestone Analysis for the ACIC Project

Let's look at a milestone analysis report of the ACIC project (other examples can be found in my earlier book⁶). This analysis, shown in Figure 11.3, was performed when the first construction iteration of the ACIC project was completed. It revealed almost no slippage in schedule and effort, and the available effort was close to the effort required until completion. Hence, the project was under control in these dimensions, and no action needed to be taken. On the quality front, however, the report indicated that the defect levels were higher than expected because of a higher number of defects injected. Hence, a defect prevention activity was to be undertaken. Furthermore, the defect injection rate for the rest of the project was revised.

Figure 11.3. Milestone analysis report of the ACIC project

The project manager reevaluated the risks and found that although the risks the project was currently facing were different from the risks at the start of the project, there was no change in the risks and their prioritization since the last milestone. The analysis report also indicated that the training activity that was planned had taken place.