Appraisals and Benchmarking | CMMI (c) Guidelines for Process Integration and Product Improvement

Many organizations find value in benchmarking their progress (e.g., ascertaining maturity level "scores," a capability level profile) in process improvement for both internal purposes and with external customers and suppliers. The organization sets its priorities based on its business and process-improvement objectives, as well as its collection of business and technical processes.

Process appraisals focus on identifying improvement opportunities and understanding the organization's position relative to the selected model or standard. Appraisal teams use CMMI to guide their identification and prioritization of findings. These findings, with guidance provided by CMMI practices, are used (by a process group, for example) to plan improvements for the organization.

For organizations that wish to appraise multiple disciplines, CMMI's integrated approach permits some economy of scale in model and appraisal training. One appraisal method can provide separate or combined results for multiple disciplines.

The appraisal principles for the CMMI Product Suite^[1] remain the same as those used in appraisals for other process-improvement models. Those principles are:

^[1] See the glossary for the definition of CMMI Product Suite.

Senior management sponsorship
A focus on the organization's business objectives
Confidentiality for interviewees
Use of a documented appraisal method
Use of a process reference model (e.g., a CMMI model) as a base
A collaborative team approach
A focus on actions for process improvement

The CMMI Product Suite provides a rigorous appraisal method for benchmarking that implements these appraisal principles. It is called the Standard CMMI Appraisal Method for Process Improvement (SCAMPI^SM).

For benchmarking against other organizations, appraisals must ensure consistent ratings. The achievement of a specific maturity level or the satisfaction of a process area must mean the same thing for different appraised organizations. Rules for ensuring this consistency are provided in the SCAMPI Method Definition Document.

Details on this method are available on the Software Engineering Institute Web site at the following URL: http://www.sei.cmu.edu/cmmi/appraisals/appraisals.html.

Appraisal Requirements for CMMI

Other CMMI-based appraisal methods may be more appropriate for a given set of sponsor needs, including self-assessments, initial appraisals, quick-look or mini-appraisals, incremental appraisals, and external appraisals.

To assist in identifying or developing appraisal methods that will be compatible with CMMI, the CMMI Product Suite provides a document called the Appraisal Requirements for CMMI (ARC). The ARC provides requirements for multiple types of appraisal methods with guidelines for determining the suitability of a particular appraisal method. Suitability addresses the accuracy and repeatability of appraisal results [SEI 0la].

The ARC document was designed to help improve consistency across multiple disciplines and appraisal methods, and to help appraisal method developers, sponsors, and users understand the tradeoffs associated with various methods. More information and a matrix detailing ARC requirements are available on the Software Engineering Institute's Web site at http://www.sei.cmu.edu/cmmi/appraisals/appraisals.html.

ISO/IEC 15504 Compatibility and Conformance

The International Organization for Standardization and International Electrotechnical Commission (ISO/IEC) Technical Report 15504 is an emerging standard for software process assessment [ISO 98].

The 1998 Technical Report version of ISO/IEC 15504 identifies two aspects of conformance: model compatibility and appraisal conformance.

For an appraisal model (e.g., Bootstrap, CMMI-SE/SW) to be in conformance with ISO/IEC 15504 (i.e., to be an ISO/IEC 15504 compatible model), it must have a "demonstration of compatibility" document to show how the model compatibility requirements of ISO/IEC 15504-2 have been addressed. These requirements are constructed to provide reasonable assurance that the model content will be reasonably broad but detailed enough to work properly with the associated documented appraisal method.

There are also ISO/IEC 15504 requirements that pertain to the actual conduct (planning as well as performance) of an appraisal. If the conduct of an appraisal is such that the requirements in ISO/IEC 15504-3 are satisfied, then the appraisal is said to be ISO/IEC 15504 conformant. One of these requirements is that an ISO/IEC 15504-compatible appraisal model be used.

The CMMI models and SCAMPI were written to support the conduct of appraisals that conform to the 1998 version of Technical Report 15504. Sponsors interested in performing an ISO/IEC 15504-conformant CMMI appraisal must provide necessary information including a demonstration of compatibility to support their needs.