Section 4.5. Techniques for Defining Other Metrics

4.5. Techniques for Defining Other Metrics

In addition to the overall metrics of throughput, efficiency, and iteration length, you will want to define metrics to measure specific aspects of your process. Definition of metrics is fraught with challenges, and poorly defined metrics can be detrimental to your organization. This section defines some techniques to help you define useful metrics.

4.5.1. Goal/Question/Metric

Goal/Question/Metric (GQM)[1][2] is a structured method that helps ensure that the metrics you gather support your improvement activities. A plethora of metrics becomes available to you when you notice the data that you are already gathering. However, it is important that the metrics activity focus on improving your knowledge or improving the process. If it does not, the numbers will quickly become overwhelming, will require significant time to analyze, and will not be valuable for the organization as a whole.

In GQM, the metrics are defined in a top-down fashion. First, you specify the goal of this metrics-gathering activity. A goal specifies four details:

• Purpose The purpose of this activity, which is usually "to improve" or "to report"

• Object (Process) The part of your system or process that is under consideration

• Issue The aspect of the object that is the focus of this activity

• Viewpoint The perspective from which measuring should occur

For example, a goal could be "To improve [purpose] the accuracy [issue] of our task estimates [object] as seen by the development team [viewpoint]."

After the goal has been set, we develop a set of questions that helps us clarify the current state of that object and its performance. These questions come from three general areas (the sub-bullets are example questions that might be developed for the previous example goal):

• How can we characterize the object with respect to this goal?

• What is the current process for making estimates?

• To what extent is the current process being followed?

• How can we characterize the attributes of the object that are relevant?

• What is the current error in our estimates?

• What is the standard deviation in our estimate errors?

• What percent of our estimates have errors that are beyond three standard deviations from the mean?

• How do we evaluate the characteristics of the object that are relevant with respect to the issue?

• Is there a trend in the error of our estimates? Are they improving or degrading?

• What level of accuracy does the development team think is an appropriate target?

After the questions are complete, appropriate metrics can be defined for each question. Note that some of the "metrics" may be subjective and based on people's opinions about a situation. Sometimes that is the best way to answer the given questions. There may be historic data in your build process, defect tracking, or time tracking systems, but do not limit the options to those sources. Be sure that you select metrics that will give honest answers to your questions. You may also find that the same metric answers multiple questions.

4.5.2. Strategy vs. Tactics

The metrics discussion in Section 4.3, "What Is the Goal?" demonstrates a critical factor in the definition of metrics: the difference between tactics and strategy. The first metrics (net profit, ROI, and cash flow) are excellent metrics for the organization as a whole and provide a strategic vision for the organization, but they need to be translated into something relevant to the work of the employees in order to allow everyone to make tactical decisions that support the strategic vision. This distinction between strategy and tactics is critical to developing useful metrics.

When management wants to set a goal for the team, they can set strategic metrics. These metrics reflect the long-term goals of the organization. However, tactical metrics that support that long-term vision must be defined on a level that is relevant to the team. In fact, you are more likely to achieve your goals if you let the team define those tactical metrics.

For example, management in a retail software company may have a vision that requires a high level of customer satisfaction and may choose to measure that through a customer satisfaction survey. The team cannot affect the results of the survey directly, but they could choose tactical metrics like defect rates and bells per release in support of that long-term vision.

If an organization has more than two levels, the highest level defines the strategic metrics for the entire organization. The tactical metrics for each level become the strategic metrics for the next level of the organization. The following diagram provides an example of how an organizational strategic goal of customer satisfaction ripples through the organization.

Figure 4.2. Strategic and tactical metrics ripple through an organization