Level 2

Selecting Metrics for Your Organization

One place to begin looking for metrics to collect that will benefit your organization is the CMMI. In the CMMI, an entire process area is devoted to metrics, that is: Measurement and Analysis. It resides at Maturity Level 2 in the staged representation, and in the Support Process category in the continuous representation. In addition, one of the Common Features Directing Implementation also addresses measurement. Just like the CMMI pre-supposes that you have standards and are using them, the CMMI also pre-supposes that you are collecting appropriate metrics about your project (e.g., schedule and cost overruns, people turnover on your project, number of errors in deliverables submitted and returned from the client, etc.). This is often not the case. In fact, in some organizations just beginning the process improvement journey, no metrics are kept. While that may shock some of you, there are large state and local agencies that have no timekeeping systems; there are some publicly held companies that do not track dollars spent on projects that are contracted out; and also the reverse, companies that only track contractor work, not their own, internal commitments.

The CMMI specifically relates its metrics to the process area (PA) activities; for example, how long it took you to arrange your planning activities (in the Project Planning PA), how long it took to write the Quality Assurance Plan (in the Product and Process Assurance PA), etc. So, what good is tracking how long you took to write a project plan if you do not also track late delivery of products and late milestones? So use the metrics mentioned in the CMMI only as a guideline when deciding which metrics might be appropriate for your organization.

We also recommend the metrics listed in the Measurement and Analysis Common feature of the original CMM for Software. We find that these metrics make sense, are directly tied to each key process area, are commonly collected and used in organizations, and are straightforward. Although it is called the CMM for Software (actually, its real name is the Capability Maturity Model CMM and most people have just added the "for Software" appellation), if you go through the book and take the word "software" out, you will find that most of the concepts and metrics described can be used in almost any organization, software-based or not.

There is another way to determine which metrics might be the most beneficial to your organization. It is called the Goal-Question-Metric (GQM) technique. We discussed this approach previously when discussing tying process improvement efforts to business goals. We basically said that this approach sounds good, but is difficult to successfully implement in low-maturity organizations because their business objectives are generally not documented in sufficient detail to really tie directly to the CMMI (their business objectives are basically described as "to make money"). However, when it comes to metrics, once an organization has seen the list of measurements it can collect, it will become apprehensive and overwhelmed. So, using a list of possible measures, and using the GQM technique to decide which metrics to select, if facilitated correctly, can be useful.

The basic G-Q-M approach is to:

• Determine the goal. What business goal are you trying to support? Why are you collecting these numbers ? What will they do for you? Why is it important to collect metrics?

• Determine the question that is most closely associated with achieving the goal.

• Determine the measurements that help answer the question, or metrics that would provide you with the response to the question.

So, a simple example of using the GQM technique is the following:

• Goal: Reduce the time it takes for our requirements definition process (while maintaining quality).

• Question: Where is the most time spent?

• Metric(s): Refer to our list below. Some suggested metrics are:

  • Number of requirements changes proposed versus number approved and number implemented

  • Time it takes for the different phases of our requirements process

  • Number of users interviewed per job function and category

  • Number of errors traced back to the Requirements phase

  • Time it takes for requirements sign-off

  • Quality of the delivered product

Obviously, more refinement of the question and metrics is needed. For example, what metric (number) can you use to quantitatively define "quality"? What part of the requirements process is broken? Do we really need to reduce the time it takes to gather, refine, document, and manage changes to our requirements? Or would it be better to reduce the time spent somewhere else in the life cycle?

Be sure to clearly define what is being measured. For example, if you are measuring the time it takes to perform the requirements process, does that only include the time to write and review requirements, or does that also include the time to attend status meetings to discuss the requirements process?

It is also critical that the purpose of each metric be explained and understood . Everyone should understand:

• Why the metrics are being collected

• How the metrics will be used

• What is expected from each person as to gathering, reporting, interpreting, and using the metrics

After selecting the metrics, pilot them in your organization. When selecting pilot projects, make sure the projects selected are already following whatever "best practices" your organization has. If not, then this project is probably not going to be successful at piloting anything. In fact, some organizations will deliberately choose a dysfunctional project to pilot improvement efforts because:

• They vainly hope that trying anything on these projects will somehow automatically improve the projects.

• They really do not want the improvement efforts to work so they can go back to doing things the way they used to.

Also make sure that these projects are not piloting everything in your organization, and that they are not going to be overburdened by collecting and piloting the metrics. There are more recommendations for planning and tracking pilots in previous chapters, as well as in the Appendices section.

Another reminder: metrics should not be collected and used to judge anyone 's work on a personal basis. That is, metrics should not be used to justify a person's salary or motivate individual performance. Metrics are used to determine how your projector organizational-level processes are working not whether your people are "good" or "bad." Dysfunctional behavior is often the result when people feel they are being measured.