Frequently Asked Questions for Planning and Conducting a SCAMPI

Using SCAMPIs in the Field

Pilots of SCAMPI v1.0 have indicated that the assessments took too much time. As a result, a performance goal of SCAMPI v1.1 was defined as follows . On-site activities should be completed within two weeks or 100 hours (excluding training and pre-onsite activities) for an assessment covering CMMI-Systems Engineering/Software Engineering version 1.1, through Maturity Level 3. Four projects would be reviewed that represent the organizational unit. Both the systems engineering and software engineering disciplines would be included.

"New" approaches taken to meet this goal included the development of the Practice Implementation Indicators (PIIs) that are collected prior to the on-site review in a PII Description (PIID). This approach is based on the best practice from many process improvement specialists, Lead Evaluators, and Lead Assessors of having the organization generate and maintain a compliance matrix showing how practices are performed in the organization. Exhibit 2 shows a sample PIID Element containing the three types of PII information collected: Direct Work Product, Indirect Work Product, and Affirmations. Direct Work Product and Indirect Work Product come from documents, and Affirmations come from interviews. This example contains information and appraisal team notes that have been collected for a Specific Practice of the Project Planning process area.

Exhibit 2: Sample PIID Element for Project Planning Specific Practice 1.1-1

Practice ID	PP SP 1.1-1: "Establish a top-level work breakdown structure (WBS) to estimate the scope of the project."
PII Type	Direct Work Product	Indirect Work Product	Affirmations
Organizational Implementation Evidence	Top-level WBS, with revision history Task descriptions Work project descriptions	Minutes of meetings at which WBS was generated or used to develop project estimate Project estimates aligned with WBS elements	Developer 1 ”" We used the WBS" Project Manager 2 ”" I worked on the WBS Team"
Appraisal Team Notes	WBS found for all projects	Minutes found for two projects

 

The initial PIIDs contain the direct and indirect work products that are prepared by the organization, and demonstrate how the practices of the CMMI are implemented in their organization via organizational documentation. In practice, often the PIID will contain the location of the work products, either hard-copy or electronic. Development of the PIIDs is nontrivial. This PIID development puts additional work on the organization to prepare for an appraisal.

The SCAMPI team is expected to conduct a Readiness Review of the PIIDs prior to the start of the on-site SCAMPI. The Lead Appraiser is responsible for giving the go-ahead at the Readiness Review to begin the on-site SCAMPI interviews. This go-ahead statement means that the SCAMPI team feels the organization is ready to have the SCAMPI team show up at the organization and begin the interview process. This step proved necessary because, in pilots, several false starts occurred, costing the organization and the SCAMPI teams time, money, and frustration.

SCAMPI v1.1 is designed to support a verification-based appraisal using the PIID to focus the team resources in areas needing further investigation. The SCAMPI team will need to verify the contents of the PIID. This is in contrast to an observation-based or discovery-based appraisal in which the team writes observations for all practices and discovers the existence of documentation through interviews and document reviews as part of the on-site review period.

The verification of the PIIDs may result in a more hostile environment being created between the organization and the SCAMPI team. The organization and the developers of the PIIDs are now stating that "X" on the PIID shows they are compliant. The SCAMPI team is now in the position of saying, "No you are not compliant because X does not match the intent of the CMMI practice."

SCAMPI also encourages mini-teams, a subset of the entire team, to interview independently. Mini-teams are given authority to reach consensus on practice implementation for each instance. The entire team remains responsible for consensus at the organizational unit level. This is a change over most previous methods that required the entire team to conduct interviews. Allowing mini-teams to interview independently may result in low-quality or inconsistent results during late-night rollups of findings. The mini-teams must be formed based on trust and experience working with the SCAMPI team members . It is not unusual for a mini-team to interview an individual, and come back and report to the rest of the SCAMPI team. The SCAMPI team may then respond by asking, "Well, did you then follow up with a question about x or y or z?" The mini-team might not have thought of those questions. The mini-team may also be allowed too much freedom. For example, the SCAMPI Lead Appraiser may allow the mini-team to interview one project manager, and another mini-team to interview another project manager, and another mini-team to interview yet another project manager. Each mini-team, if not immediately consolidating their results with the entire team, may find that the first project manager does something in agreement with a practice, the second one does not, and the third manager was not even asked a question relating to that practice. During the late-night, last-night on-site rollup of results to produce final findings, the team may be surprised to learn that is has inconsistent findings on a number of practices. So, while the concept of mini-teams sounds like it will promote time-savings by allowing more people to be interviewed in a shorter timeframe, the mini-teams must be managed closely by the Lead Appraiser ” which puts yet another burden on the Lead Appraiser.

Some of the older assessment and evaluation methods had more specific guidance in some areas; for example, the number of focus projects and the assessment team size . This guidance is not as clear in the ARC and the SCAMPI. This leads the user of these new methods to make more decisions. And while one of the stated purposes of the ARC and the SCAMPI is to ensure more consistent results in assessments, this lack of guidance may, in fact, result in less consistent results.