25.2 Metrics for Controlling Configuration Management Performance

In Principles of Software Engineering Management , Tom Gilb says that everything can be made measurable in a way that is better than not measuring at all. One of the principles for good planning is to define specific and measurable goals for activities. But it's not enough for goals to be measurable: metrics must be defined, and measurements collected and used. This section provides a brief discussion of metric-related issues and gives examples of metrics related to configuration management.

Table 25-1 defines several terms related to measurement and metrics. To give an example, the metric for the size of a book is "the number of pages"; the measuring method is "look at the page number on the last numbered page"; the plan is "the reader when she starts on the book"; and the measurement for this metric for Alice in Wonderland in a certain edition is "54."

Table 25-1. Metric and Measurement Terms

Term	Meaning
Metric	The definition of what you want to measure
Measuring method	The description of how you are going to measurehow you are going to collect the measurements for the metrics
Measurement plan	The description of the metrics, measuring methods , who is going to measure, and when, and how the measurements will be analyzed and used
Measurement	The values collected for the metrics

Metrics in General

Metrics are the definitions of what you want to measure, both directly and by calculation. You can collect or produce a number of data types, as shown in Table 25-2. Data may be objective, with no personal evaluation involved. These are often enumeration, and tools may be used to collect the measurements. Conversely, data may be subjective , involving an element of evaluation. This entails some uncertainty, but the data is often cheap to collect, and its usefulness should not be underestimated. Metrics may well be simple. Nothing is wrong with a metric for which the measurement is a verbal answer of yes or no. This could be, for example, whether a convention is used or not.

Table 25-2. Data Types

Data Type	Description
Raw data	Collected data (e.g., time sheets or event registrations)
Direct measuring	A single feature extracted from the raw data (e.g., the number of events and the collection time interval)
Indirect measuring	Calculated features from direct measuring (e.g., event frequency = events/collection time)

It's important to agree on certain aspects when defining metrics, as shown in Table 25-3.

Table 25-3. Aspects of Metrics

Aspect	Description
Definitions	Examples: lines of code (new, changed, removed, comments, blank) and importance (15 or 51)
Units	Example: hours or seconds
Consistency	For suites of metrics that in total are to provide one answer

When defining metrics it's important to understand the various scales that may be used and what you can do with metrics in various scales. The most-used scales are listed in Table 25-4. Whatever you do, don't mix apples with oranges, unless you want to make fruit salad.

Table 25-4. Scale Definitions

Scale	Characteristics	Statistics	Examples
Nominal	Undefined order = No arithmetic	No average, median, or variance	Classifications, like language (Pascal, C, C++) Problem type
Ordinal	Defined order =, >, < (monotone) No arithmetic	Median, but no average and no variance	Ordered data set, such as capability levels Scales (low, middle, high)
Interval	=, >, < Addition, subtraction	Average and variance	Dates and time Degrees Celsius or Fahrenheit
Ratio	All arithmetic may be used Includes 0	Average and percent deviation	Time intervals Degrees Kelvin Length, height, cost
Absolute	Count		Number Probability

Measuring Methods

The measuring method is the description of how you are going to measure. Measuring is assigning a value to a metric. Table 25-5 shows a number of qualities to consider when defining measuring methods, which should be as simple as possible.

Table 25-5. Requirements for Metrics

Measurements Should Be	Meaning
Repeatable	Example: same measuring time and same instrument
Precise	Valid scale and known source
Comparable	Example: over time and/or between sources
Economical	Affordable to collect and analyze compared to their value

Measurement Plan

A measurement plan defines metrics and measuring methods. It describes who is going to measure, and when, and how the measurements will be analyzed and used. The measurement plan is individual from company to company.

When planning, ensure that the defined metrics answer questions to which you want answers. Don't measure for the sake of measuring. Also, collecting the measurements should be as simple as possible. Ideally, all measurements should be producible from already registered information, so that collection doesn't entail extra work. Maybe available data just needs to be used in a new way.

Not only should measurements be used; the usage should be visible for those who provide them. So only measure what will be used immediately and will provide quick and precise feedback. The plan must also create confidence. Measurements from process usage must never be used to punish or reward individuals. Remember: unwise measuring may induce a lot of strange behavior in people.

Examples

A number of techniques describe the process of selecting and defining metrics for configuration management, such as GoalQuestionMetrics (GQM) or Goal Driven Software Measurement. Following is a list with suggestions for metrics that may be used to analyze how configuration management is performed. The list focuses on configuration management processes, not other processes and not the product. It is by no means exhaustive but may serve as inspiration. Metrics may be

• The number of registrations of identified items made, maybe by item type

• The time interval in which the registrations have taken place

• The time used for registration, maybe also by configuration item type

• Events in connection with registrations, maybe by type and/or configuration item type

• The event rate for registrations, such as number of erroneous registrations per hundred

• The average time for registration, maybe by configuration item type

Identical metrics may be defined for

• Placement in storage

• Extract for use

• Extract for production

• The handling of event registrations

• The handling of change requests

• The completion of milestones defined in the configuration management plan

Metrics may be defined including cost, such as the cost of the activities. Measurements may show new aspects of things you thought you knew everything about.