Section 4.6. Deploying Metrics

4.6. Deploying Metrics

4.6.1. Team vs. Individual Metrics

One of the critical things to watch as you deploy metrics is whether you will measure the entire team or individuals. Remember that the performance of the team as a whole determines the team's success level, so we must gather team metrics. Most agile organizations only measure at the team level because the cohesiveness of the team is critical to its success and applying metrics at an individual level can undermine that cohesiveness.

If one member is not performing, the team will recognize that situation without metrics and should manage it themselves as much as possible. Often, applying metrics at an individual level can be seen as punitive and arbitrary. For example, if we used throughput to measure an individual, the metrics may imply that the people who took on the challenging tasks were not performing well. Although that example seems extreme, very few metrics capture the value of an individual team member accurately. I know of a company that measures individual engineers by lines of code produced, presumably with the goal of increasing the amount of software delivered. This is an excellent example of an arbitrary metric that engineers will resent and that will not achieve its goal. In addition to discouraging refactoring (which could reduce the total lines of code), this metric affected the tasks that engineers wanted to work on (easier tasks that required only copy-and-paste code were preferred). The result was tension within the team.

When you use a metric to measure an individual and that metric does not reflect that person's role in the organization, the metric is seen as arbitrary and is unlikely to cause the desired behavior. In the team that was measured by lines of code, the senior engineers who spent a good portion of their time mentoring the junior engineers and guiding the architecture of the system were unfairly punished by this metric.

However, all caveats aside, most organizations require some performance measurement on an individual level for annual performance reviews. If you must have a numerical or objective measurement of an individual, do not use the same metrics you are using to measure the team. Instead, let the individual help you define the metrics that will be used to measure him. For example, an engineer who is aspiring to technical leadership may want to be measured by the number of design decisions he participates in or the complexity of the tasks he undertakes. Another engineer who wants to expand his knowledge of the entire system may want to be measured by the number of classes he modifies throughout the year. In both of these examples, the individual metrics are tactical in nature, and the process by which you define these metrics can ensure that they support the team's strategic metrics of throughput, efficiency, and iteration length.^[10]

^[10] Note that metrics that were tactical for the team in support of the organization's strategic metrics have now become the team's strategic metrics with individual tactical metrics. This cascade of metrics can flow through many levels of your organization. The top layer defines strategic metrics, the next defines its tactical metrics, and those tactical metrics become strategic metrics for the next level down the organization.

4.6.2. Side Effects and Risks of Metrics

Definition of metrics requires careful consideration because metrics can have unexpected side effects. For example, one organization I worked with measured engineers by the number of defects they repaired. The goal was to reward the engineers who fixed problems. However, defects were often assigned to the engineer who developed the code. This meant that the metric inadvertently rewarded the engineers who created defects.

When you define a metric, you have to consider the behavior that would maximize or minimize its value to be sure that you will achieve your goal. In addition, be very careful about the interactions between metrics. There are times when improving the value of one metric will cause the value of another metric to degrade. Often letting the team set its own tactical metrics can eliminate these situations. Given strategic goals, the team knows best how to measure their performance against those goals, and if they understand the motivation for the metric, they are more likely to behave in support of the goal.

It is also important to be very careful about your definition of a metric. It must specify how the data will be gathered and analyzed to compute its value. This specification should also include the rate at which the metric will be measured. For example, if it will be part of a quarterly report, specify that the data will be gathered weekly and that the report will contain the data available one week before the report is complete. This will help ensure consistency of the metric's value over time and across the organization if necessary. Consistency will be critical because you will want to evaluate trends in these numbers so that valid conclusions can be drawn.

The underlying goal of any metrics activity is improving performance. Because metrics will drive decisions, it is critical that you define them carefully, gather them consistently, and watch for unexpected side effects. As with all agile activities, you don't define metrics once and leave them for eternity. You should always be watching those definitions and adjusting them as your organization changes.