Training Programs

Measurement Program

If the organization has not developed an integrated measurement program, it is likely that measures ^[2] and measurements will be unbalanced across multiple process improvement initiatives. Different improvement models place different levels of importance on measurements. To exploit some of the resulting differences, we offer the following questions and suggested approaches.

Questions you should ask regarding your measurement programs include the following.

What measures are being collected for each discipline?

Some measures can cover several disciplines, for example, effort measures and earned value. Other measures would be unique to a discipline, such as number of late deliveries from vendor X or lines of code per hour .

Are there measurement specifications defined for all measures?

A measurement specification should contain a definition of the measure, source of the data, collection mechanism, criteria for counting, unit(s) of measure, and expected range value. In addition, guidelines on interpreting the measurement information, including an analysis approach and criteria for decision making, should be documented. Why do we need all this stuff? Well, measures are like many things in life; "if it ain't written down, it ain't so." For example, when you ask someone, "How many requirements do you have in your system?", the only way to make sense of the answer, and for that matter the question, is to have a definition of what a requirement is.

What measures are being presented to management on a regular basis?

The approaches to presenting measurement to management vary widely across organizations. We have worked with groups that hold a formal monthly measurement review with senior management and we have worked with groups that simply provide measures in quarterly project status reports . The key to success with measures is to be provide accurate measurement data to management on a regular, periodic basis during both good times and bad showing both the good news and bad.

How mature are the measurement systems for each discipline?

Many of the organizations we work with began their measurement initiatives in the software area and did not place as much emphasis on other disciplines. Therefore, software measurement in these organizations is often more mature. ^[3] Measurement maturity shows up in several ways, such as how well measures are defined, how consistent measurements are taken and used, and how well the actual data collected reflects the processes being executed.

You will want to understand these issues and the success individual disciplines have had in your organization in order to share best practices across the organization. Some organizations have software data identifying critical components , attributes, and measures of the standard processes based on critical business issues. If that is the case, showing the value the organization has gotten from more mature software measures may be helpful in getting buy-in from the other parts of your organization that have not been as intensely involved in process improvement or are just starting their own process improvement initiative.

How are the measures collected within the disciplines?

Again, we often find wide variation in how measures are collected across the disciplines. For example, you might use a manual method for calculating and collecting critical performance factors for formal reviews, or you might have a fully automated, integrated development and time tracking environment that produces weekly and on-demand measurement charts. These charts may include effort, earned value, requirements changes, size (code and documents), number of reviews, number of defects, and productivity measures. As a rule, you should make collecting measures as painless as possible. Remember that "if it is too damn hard, people won't do it." Automate data collection as much as possible, but do not forget to verify, validate, and monitor collection to be sure you are getting the right stuff. A brain is required to analyze and comprehend information, and a brain is required to ensure that the data collected makes sense. An automated tool does not have a brain.

Following are some suggested approaches to get answers to the questions.

Collect and review the related documentation

The first thing you need to do is collect all the documentation related to measurement policies, processes, and procedures. Some of this may be embedded in documentation for project planning and tracking or other process areas. Some disciplines will have developed measurement handbooks containing measurement specifications. Collect whatever you can find.

Collect examples of measures presented to management

Collect copies of status reports and briefings prepared for and presented to management. These include status from projects, major programs, functional areas, and the entire organization or division. Identify which measures are being reported and how often.

Identify all the groups (or individuals) with measurement responsibility

You need to know which groups and individuals are actually doing jobs related to measurement. You may find groups within individual disciplines, for example, a Process Action Team for Measurement chartered by the SEPG. You may find groups or individuals responsible for collecting and maintaining time-reporting data. You may find groups or individuals responsible for maintaining data used in bid and proposal activities. Go through the questions in Exhibit 1 and answer them for the groups and individuals with measurement responsibility.

Identify how the measures are being collected

You may find this information in a measurement specification. You may find this as part of the status report. You may have to interview the individuals responsible for the measures to see how they really do it. It is not unusual to find people "guesstimating" guessing at a number and calling it an estimate, or worse , presenting it as actual data. ( Note: We do not encourage this behavior; however, we recognize its existence.) You need to be somewhat understanding and flexible during this data collection phase. In some cases we have had to get senior management to declare an amnesty for all past behavior.

Create a measurement log

Create a measurement log summarizing the information identified above. At a minimum, this log should contain the measure, how it is documented, group or individual responsible for collecting it, tool or method used to collect it, and how often, to whom, and how it is reported.

Our experience is that if you go through all the steps above, you document your results, and review your findings with management, you will be able to identify the business case for merging your measurement programs, collection systems, and measurement databases. The primary business reasons that we have found for integrating measurement programs are removing duplication of effort, leveraging best practices, and establishing consistency.

^[2] Measures in this section refer to both base and derived measures. Base measures are simple values of some attribute, for example, the size of a document in pages or effort to produce a document in hours. Derived measures are defined to be a function of two or more base measures, for example, productivity in hours per page to produce a document.

^[3] We really do mean many, but not all. We have had the opportunity to work with both systems and acquisition program that exhibit high measure maturity but these are the exceptional cases; and while it may be politically incorrect to talk about software success and systems and acquisition, we think it is important to share our experiences.