Monitoring Project Performance

Project performance can be monitored in a number of ways. This section briefly mentions one inherent in the RUP and then discusses another, Earned Value, in more detail.

Releases

The RUP, due to its iterative nature, provides the best opportunity to observe real project status through demonstrable, executable releases. I encourage outsourcing organizations to work with their contractors to determine the appropriate time to examine these releases. These releases amount to snapshots in time of the state of the actual product being produced. Having the people who will use the project evaluate the executable releases is the best way to measure progress.

Earned Value

Earned Value has become an increasingly popular technique for tracking true project performance. Outsourcing organizations sometimes ask, or even require, their contractors to use Earned Value techniques to help them monitor a project's progress. To understand what Earned Value is, examine Figure 3-3. Is the scenario shown in Figure 3-3 a good situation? At first glance, it would seem so. Figure 3-3 shows that the project is running under its planned budget, which would seem to be a good situation. But in reality, you cannot tell if the project described by Figure 3-3 is running well or is having problems, because you cannot tell how much work has been done. If very little work has been accomplished, the project may be in serious trouble even though it is running under budget. On the other hand, if the work to be completed by the point identified by Time (x) has been completed successfully, the project is in excellent shape. Thus, Earned Value is useful because it integrates the number of resources (time and budget) with the amount of work actually performed.

Figure 3-3. Is this project running well?

Derived from "Earned Value, Clear and Simple," used with permission from Primavera Systems, Inc.

Figure 3-4 illustrates some of the fundamental metrics that are computed as a part of Earned Value analysis.

Figure 3-4. Fundamentals of Earned Value management

Ibid

Most projects using Earned Value require the metrics illustrated in Figure 3-4 to be calculated each month. To properly calculate Earned Value, the contractor must track hours spent by project staff at a higher level of granularity than they may be accustomed to. This requires the contractor to closely track not only the total number of hours each person spends on the project, but also how many hours each person spends on each major task defined in the project's Work Breakdown Structure (WBS).

Monitoring Earned Value is quite useful to ensure that the project stays on task. If projections of cost and schedule begin to move significantly beyond budgeted amounts (10% is a reasonable figure), this does not necessarily mean that the project is in trouble. It means that something is happening on the project that requires investigation. The fluctuation may be due to a change in project scope or an unscheduled event that occurred. The contractor should discuss this with the outsourcing organization and decide whether corrective measures are warranted.

Points to Consider When Using Earned Value on Software Projects

The use of Earned Value techniques is not without controversy. Earned Value is more appropriate on projects involving a predictive lifecycle model, such as Waterfall-based projects. It is also common in the manufacturing and construction industries, where it is easier to quantify certain variables, such as the cost of materials. This does not mean it cannot be used on iterative lifecycle projects. It does mean that additional aspects must be considered. For example, to track and calculate Earned Value, you must estimate the value of each task. The values of all these tasks add up to the project's total cost. As the project progresses, the percentage of completion of each task is recorded, along with the funds expended. This, together with the budgeted amount of each task, allows you to calculate the Earned Value for the task.

One challenge with using Earned Value on software projects involves the Budgeted Cost of Work Scheduled (BCWS). This figure is based on rough estimates or guesses made early in the project's lifecycle, particularly on Waterfall lifecycle projects. As noted earlier in this chapter, these figures may be completely unrealistic. As a consequence, the result of the calculation of Earned Value in this situation is questionable.

Another challenge using Earned Value with iterative projects is that the scope of the development tasks may change. As iterative projects progress, new requirements may be discovered and added during the project, or the existing requirements may change. Finally, the knowledge gained as the project progresses might lead you to conclude that the earlier BCWS figures were wrong. This changes the budgeted cost of some of the tasks. As a result, the Earned Value calculation result can be misleading.

One possible way to make Earned Value useful and to integrate it with an iterative lifecycle is to calculate it for each individual iteration:

• Calculate the budgeted cost of an iteration right before it begins. This should be easy, because iterations are time-boxed. You can calculate the cost simply by taking the number of hours of labor performed for the iteration by each person and multiplying it by the person's hourly rate.

• Determine the estimated effort for each requirement or feature that is planned for the iteration. You should already be performing this activity in some capacity to determine what is reasonable to expect to accomplish within the iteration.

• As the iteration is conducted, individual features or requirements are marked complete only after they have been successfully tested.

• If you complete exactly what was assigned to the iteration and finish at the end date of the iteration, the cost and schedule variances should both be zero.

At the end of the iteration, calculate the actual Earned Value. You can also calculate a cumulative figure using the results from the earlier iterations. These results will give you useful data for consideration. The advantages of calculating Earned Value in this way are as follows:

• The BCWS becomes more accurate with each iteration. As the project progresses, the staff becomes more familiar with the problem domain, as well as the environment. Estimates made at the beginning of each iteration improve with each iteration.

• The criterion for Earned Value is tied directly to an executable, observable, and verified unit of functionality. Thus, you calculate the percentage complete by dividing the number of verified, tested requirements or features by the number planned for the iteration. Therefore, this is not a subjective estimate.

Projections for the remainder of the project can be made from the iterations completed up to the date the projection is made. This technique also stresses the importance of prioritizing requirements.

Here are some additional points to consider using Earned Value:

• Most decisions to use Earned Value are driven by the outsourcing organization. In fact, some organizations mandate its use. However, it is less meaningful on Firm Fixed Price contracts, since the burden and risk of delivering the product within budget fall entirely on the contractor. In that case, the budget data, while interesting, does not provide significant value to the outsourcing organization. If the outsourcing organization still wants to use Earned Value on this type of project, the schedule variance information should be the prime focus.

• The most tedious part of employing Earned Value is collecting the data. Care should be taken to choose tasks at a high-enough level for collecting data so that the staff is not overwhelmed with a flood of different charge codes to use. Software developers are not likely to accurately track their time if they must use more than two or three charge codes per day.

• If the organization requires staff to fill out time sheets, it is helpful if charge codes can be set up for each specific task on the project directly in the time sheet. In other words, don't require staff to fill out one time sheet for calculating Earned Value and another for the organization to which they belong. The staff will become frustrated at having to keep track of time data in two separate places.

For more information on Earned Value, refer to the references in the bibliography, especially the book Earned Value Project Management, Second Edition, by Quentin W. Fleming and Joel M. Koppelman.