Establishing a Metrics Program

The importance of measurement and metrics is hard to overstate. Measurement provides critical insight to management, allowing management to support strategic decision-making processes. Measures are numeric values assigned to a given artifact, software product, or process. A metric is a combination of two or more measures that together provide some business-relevant meaning. For example, when considered separately "lines of code" and "number of security breaches" are two distinct measures that provide very little business meaning because there is no context for their values. A metric made up as "number of breaches / lines of code" provides a more interesting relative value. A comparative metric like this can be used to compare and contrast a given system's "security defect density" against a previous version or similar systems and thus provide management with useful data for decision making.

Ideally, metrics and measures will focus on four primary areas: project, process, product, and organization. The first three are specific to a given artifact or activity in a software development effort, while the purpose of the latter is to determine trends across the three other areas.

Establishing a metrics capability is a challenging undertaking. Early standard software process approaches focused on sequentially building a level of sufficiency in four areas and in a particular order: process, controls, metrics, and improvement. Unfortunately, following these basic steps in the prescribed order implies that metrics are not addressed until late in the program. By then it may be too late. In this case, processes and controls put in place early may not be properly designed to provide the kinds of metrics that are needed later. In those cases, some significant rework may be required to achieve business alignment.

All metrics should render decision criteria based on strategic business objectives. For that reason, business objectives must be articulated first and used to guide the entire program, from process and control development onward.

A Three-Step Enterprise Rollout

Figure 10-1 shows a simple three-step rollout plan for establishing an enterprise-wide, metrics-based software security program. This approach can be adapted for use in rolling out any large initiative. The three fundamental steps are (1) assess and plan, (2) build and pilot, and (3) propagate and improve.

Figure 10-1. A three-step rollout plan for enterprise adoption of software security best practices, based on establishing clear measurements and metrics up front.

Step 1 involves getting a handle on the current state of the business. This includes understanding the goals of the program writ large, collecting data to assess current state, and then comparing current state to goal state. This is in some sense a gap analysis. As an example, rollout step 1 in a large software security program will include understanding the in-place SDLC and assessing how well it covers the touchpoints discussed in this book. If code review is currently practiced only at the unit level by developers who are not using a static analysis tool, a clear gap has been identified between the goal (state-of-the-art software security) and reality. Since there are likely to be a large number of application development projects underway simultaneously in any large company, developing a rating system to assess each project is an important part of baselining. This leads to a measurement and metrics regimen that can be evenly applied throughout the rollout. Note that some of these measures can be taken from software artifacts as briefly described earlier.

Counterintuitively, it may be best to begin rollout step 2, build and pilot, by identifying a software project that is ahead of the game. That is, because you want to maximize the possibility of pilot success, starting with the project with the smallest gap may make things easier. For example, if the software project chosen for pilot is already using static analysis tools for reliability (looking for null pointers and other simple bugs), adopting a security-related source code analysis tool is likely to be fairly straightforward for the project team. Because we have a set of measurements in place, we can assess progress over time in rollout step 2 and refine our measurement system. Note that in almost all cases, training programs will need to be developed that clearly describe both the goals of the improvement program and how to actually carry out the new best practice. The material in Part II of this book should be particularly useful.

A successful pilot program provides an excellent real-world case study of the adoption of a best practice in one part of the enterprise. This success story provides "proof in the pudding" that a best practice, like code review with a source code security scanner, can be successful in the organization. Rollout step 3, propagate and improve, involves taking the best practice wide. By relying on the baseline gap analysis results from step 1, we can logically approach the problem of wide adoption. Our measurement program helps keep tabs on progress and is extremely useful in alerting us of adoption issues as they crop up. The training program developed in step 2 is a critical part of the widespread adoption of a best practice in a company. Also helpful is an information portal for software professionals to use as a resource as they adopt various software security touchpoints.

By following this straightforward rollout plan, a very large organization can transform its existing SDLC (or more likely SDLCs, plural) with the addition of best practices for software security. The idea of an SDL is thereby a combination of your SDLC with the software security touchpoints. This key point bears repeating. Presumably your organization already knows how to make software and has already been shipping it for years. There is no reason to throw out everything you're already doing and start from scratch. Instead, your already-in-place SDLC can be adjusted by adding touchpoints. My process-agnostic approach, based around software artifacts, makes this possible.