Notes

Frequently Asked Questions for Planning and Conducting a SCAMPI

The following questions are the most frequently asked questions that we have been asked to address. We have pulled them from lists of questions that prospective clients for SCAMPIs have sent us.

1. How will you determine the number of focus projects to include in an appraisal?

The primary criteria are that the focus projects are a representative sample of the organizational unit's overall behavior, provide at least two instances of the processes being investigated as sources of objective evidence, and provide representative coverage of the life cycles in use within the organization.

Our goal would be to keep the number of focus projects small (usually four). If the scope of your appraisal includes systems engineering and software engineering, or you have a large organization, it may be difficult to meet the stated criteria with four projects. Appraisals of very large organizations have required over 10 focus projects.

2. What is the minimum number of projects (we have three core businesses)?

We consider the minimum to be four projects; however as stated above, this may be difficult to accomplish. For example, if the three core business units use significantly different processes, methods , and tools to produce their products or provide their services, this could expand into a need for considering 12 projects.

3. Do you consider the organization as a separate project?

We do not understand exactly what this question means. As far as planning goes, we will need to interview organizational managers ” middle managers responsible for technical development, process improvement, training, quality, and measurement.

4. Can projects that are partially through the product development life cycle be included in the appraisal? How many projects will have to provide full life-cycle coverage?

Yes, most projects should be ongoing and therefore are most likely partially through the product development cycle or some phase or iteration. There is no exact answer to this question ” the selected projects need to provide representative coverage of the life cycles in use in the organization. That is, the sum of the selected projects' life cycles needs to cover the organization's life cycle.

5. How is model coverage determined?

If by "model coverage" you are referring to verifying and validating objective evidence, this is determined by reviewing documents and artifacts and affirmations for each practice.

If by "model coverage" you are referring to reference model scope, this is determined by the Lead Appraiser reviewing appraisal goals with the sponsor and ensuring the sponsor makes an informed choice.

6. Do you characterize implementation of model practice instantiation on the basis of specific and generic practices, or at the subpractice level?

Appraisals consider three categories of components as defined by the CMMI:

• Required: specific and generic goals only

• Expected: specific and generic practices only

• Informative: includes subpractices , discipline amplifications, generic practice elaborations, and typical work products

So, to be compliant with the model, we would characterize implementation based on specific and generic practices and use subpractices to guide interpretation of the practices.

7. Are typical work products, discipline amplifications, or generic practice elaborations listed in the model included in the instantiation decision?

Yes, as stated above, these are informative and would guide interpretation of the practices.

8. Can you conduct a two-week, eight- hour -a-day, on-site CMMI Target Profile 5 appraisal, covering systems engineering, software engineering, and supplier sourcing with six appraisal team members (including the lead appraiser)? Please be specific.

No, based on no additional knowledge of the organization and the time it takes to conduct a SCAMPI. Our estimate for a SCAMPI Class A Appraisal covering 24 process areas through Capability/Maturity Level 5 with six appraisal team members that have received Team Training to include Introduction to CMMI, SCAMPI Appraisal Team Member Training, and High Maturity Assessments is 13 days. In addition to the Lead Appraiser, we base our estimate on at least one additional external appraiser on the team. Even with 13 days, it is probable that some days may be longer than eight hours.

Only when we have worked with the organization and have a better understanding of the organization's processes, organizational structure, and qualifications of the appraisal team members would it be possible to decrease the proposed schedule.

9. How many participants do you plan to interview for an appraisal covering software engineering, systems engineering, and supplier sourcing? Please use a matrix to indicate , by function, the total number of staff involved, and estimate the total number of labor hours (e.g., project manager, middle manager, test, quantitative measurement specialists, etc.).

Because we have not performed detailed planning and do not have an understanding of your organization's processes, roles, and structure, this is very difficult to answer with any certainty . However, Exhibit 3 will give you an idea of what to expect.

Exhibit 3: What to Expect

Role	Estimated Number of Interviews	Hours per Interview	Number of Interviewees per Interview	Estimated Total Hours
Project managers	4	2	1	8
Measurement group (assumes separate organization and project groups)	2	1.5	4	12
Middle manager groups (assumes separate systems and software groups)	2	2	4	16
Training group	1	2	3	6
Requirements representatives	1	2	4	8
Development representatives	1	2	4	8
Test representatives	1	2	4	8
Integration representatives	1	2	4	8
Process improvement group (assumes separate leads and specialists)	2	2	4	16
Quality assurance group (assumes separate systems and software)	2	2	4	16
Supplier sourcing group	1	2	4	8
Tool group	1	2	3	6
Estimated total				120

 

10. How many follow-up interviews do you anticipate? Please estimate the level of effort (e.g., total number of labor hours).

Our goal is to have nearly no follow-up interviews. For planning purposes, you may want to consider allowing time for two one-hour follow-up interviews with four people each for a total of eight labor hours for the interviewees.

11. Do Lead Appraisers interpret the CMMI practices as required or expected? Please elaborate. Under what conditions is an "alternate practice" acceptable in satisfying the CMMI?

Specific and generic practices are expected model components. Expected components are practices an organization will typically implement when it is trying to satisfy a set of specific and generic goals. Ordinarily, an alternative practice has to contribute an equivalent effect toward satisfying the goal associated with the practice. In some cases, an alternative practice may not explicitly contribute an equivalent effect because a project's characteristics are different from the characteristics assumed in the model. An example of this might be where the project's customer performs requirements management. The alternative practice might be to review the customer's requirements management process and activities. The practices implemented by the project in this example might not contribute an equivalent effect toward the goals. The decision as to whether an alternative practice is acceptable is an appraisal team decision. The Lead Appraiser's role in that decision is to ensure the integrity of the SCAMPI process and help the team correctly interpret the model.

12. How do Lead Appraisers interpret "institutionalization" of a process area? Is it expected that all generic practices be performed? How is the generic goal judged in the absence of one of more generic practices? Is there any required time duration that a practice must be performed to consider it institutionalized?

Institutionalization in the CMMI is interpreted as satisfying the requirements of the Generic Goals (GG). For the staged representation, this means that GG 2: Institutionalize a Managed Process is satisfied for Level 2. In addition, GG 3: Institutionalize a Defined Process is satisfied for Level 3.

Yes, we expect generic practices to be performed. If one or more generic practices are not performed, the assessment team will determine the impact of the missing practice upon satisfying the required generic goal. Although each generic practice is expected to be performed, the implementation of the generic practices to individual process areas is often not distinguishable (one process area versus another), and they may appear to be absent for a process area or areas. This may be acceptable but the assessment team would have to understand and determine whether the implementation is reasonable.

While the SEI has not published specific guidelines for institutionalization, a number of appraisers consider a minimum of six months as a demonstration of basic institutionalization. Other appraisers expect to see that practices have been performed through several cycles. And other appraisers have considered the requirement to be a demonstration that the practices have been performed in the past and will continue after the appraisal. In addition, some Lead Appraisers consider how often a process can be performed. For example, on a 20-year-old legacy system, they would not consider it reasonable to expect that the initial customer requirements were captured using today's process; however, they would expect new requirement changes to use today's process. With no stated institutionalization requirement, a Lead Appraiser and Appraisal Team may consider any or all of the above guidelines.

13. How do Lead Appraisers interpret the need for documented procedures in the CMMI, now that the phase " according to a documented procedure" is not used? Are documented procedures required, or can an acceptable alternative practice be to perform the practice without a documented procedure?

For any process area to satisfy Level 2 "GG 2: Institutionalize a Managed Process," a Lead Appraiser would expect that "GP 2.2: Establish and maintain the requirements and objectives, and plan for performing the process" is performed. Although documenting the process is only a subpractice of GP 2.2, it is quite implicit in the statement of GP 2.2 and hence is expected. The clearest way to meet the expectations of GP 2.2 is to have a documented process and plan covering the process area. An alternative practice of "just doing it" is generally not acceptable. The term "plan" in the CMMI has been expanded to include such documentation as processes and procedures.

Part of an acceptable process description is that it contains an appropriate level of detail. In some areas of the process, this means that a documented procedure is appropriate, but neither "expected" nor "required" in CMMI terminology. Note that there are only a very few mentions of "documented procedures" in the informative parts of the CMMI model. The test that the SCAMPI team will have to apply is not whether there is a documented procedure for "X," but whether the process, the plan, or supporting procedures are adequate in the area of "X." This is the same for both the project- and organization-level processes.

For any process area to satisfy "GG 3: Institutionalize a Defined Process," a Lead Appraiser would expect that "GP 3.1: Establish and maintain the description of a defined X process" is satisfied. The expectation is that a process from the organization's set of standard processes is tailored to address the needs of a specific instantiation. As defined in the Model Terminology section of the model, the phrase "Establish and Maintain" connotes a meaning beyond the component terms; it includes documentation as well as a usage component ” that is, documented and used throughout the organization.

14. In which situations is the Supplier Agreement Management process area applicable/non- applicable ? Please address Integrated Product/Process Teams ("badgeless" teams of multiple contractors), tool vendors , and hired labor ("bodyshop workers"). Please address customer-directed subcontractors .

The Supplier Agreement Management process area states that it is applicable to a "contract, a license, or a memorandum of agreement" for acquired products that are "delivered to the project from the supplier and becomes part of the products delivered to the customer." It applies primarily to both internal and external "arm's-length" relationships where the project manager is not directly involved in managing the activities of the supplier.

The Supplier Agreement Management process area does not apply to products that are not delivered to the project's customer, such as nondeliverable development tools. It does apply to commercial vendors if their product is included in the product delivered to the customer. It does not apply to hourly contract workers who are managed in the same way as any other member of the project team.

For all deliverable work not covered by the Supplier Agreement Management process area, the project management process areas (PP, PMC, ISM, RSKM, IPM, and QPM) and the IPPD process areas (if IPPD is implemented and that model is used) will apply.

15. What are the minimum contents of an organization's set of standard processes?

The OSSPs (Organization's Set of Standard Processes) must cover the processes for the engineering, project management, engineering support (e.g., QA and CM), and organizational activities (e.g., training and process group ” limited to software if the assessment is limited that way). The OSSPs describe the fundamental process elements that will be part of the project's defined processes. It also describes the relationships (e.g., ordering and interfaces) between these process elements. Often, the OSSPs for the organizational processes are one and the same as the defined processes. The OPD process area provides specific information as to what might be in the process descriptions and how they might be defined (see Chapter 6). This information is reasonable for most organizations and would be considered by the SCAMPI team. However, there are no hard and fast rules as to the minimum content.

16. Some projects' defined software processes may pre-date the establishment of the organization's standard software process. Is a matrix showing the traceability sufficient?

It is expected that the majority of the organization's projects would be using the OSSP. However, an organization that is improving its OSSP is likely to have some number of long-term projects that pre-date the establishment of the current OSSP. For business reasons, these projects may not migrate to the new OSSP. All projects' defined processes should, however, be recognized as alternative processes by the organization. A Lead Appraiser would expect to see rationale for why projects were exempted from using the OSSP. A Lead Appraiser would expect to see an analysis of the existing process against the OSSP that may take the form of a traceability matrix. However, the existence of such a matrix would not in and of itself be sufficient to exclude a project from using the OSSPs. Furthermore, it should be recognized that a project's defined process that is not tailored from the OSSPs will have an impact on organizational learning and any efforts to establish performance baselines for the organization and for these projects. The organization would have to make appropriate accommodations for these shortfalls.

17. Some of our projects require special security clearances for access to any project documents. A subset of the internal assessment team members has these accesses . What is your approach for assessing such projects?

If the issue is limited to gaining access to the project documents, or if this also includes limiting the access during interviews, then we need to discuss this situation further. This represents a high risk that the team will not be able to achieve consensus on some process areas, that sufficient data collection may not be performed, and certain process areas would have to be "not rated." "Not rated" does not satisfy SCAMPI rating requirements and will result in the organization not receiving a level rating. What we have done in the past is make sure that all members of the team have the appropriate security clearances. Team members could also sign a nondisclosure agreement.

18. What metrics are required for each process area? Must separate time accounting be performed for every process area?

There is no simple list of required metrics for each process area (see Chapter 17). The requirement is for sufficient metrics to monitor and control the processes at Level 2, sufficient metrics to support the organization's needs at Level 3, and sufficient metrics to achieve reasonable predictability of the project's processes (relative to the critical business issues) at Level 4. Metrics should not be defined around CMMI process areas; they should be defined around the project's and organization's process and their need for metrics. Separate time accounting is not required for each CMMI process area. Hours to perform the individual processes is often not very useful; however, many organizations provide this metric because it is easy compared to collecting more interesting metrics. Some organizations at a lower maturity level also find this metric initially helpful in reviewing the time spent and dollars expended in doing process improvement work. It helps them build a preliminary case for return on investment (ROI).

19. How is the generic practice for collecting improvement information interpreted? What is the minimum information that must be collected?

This generic practice represents the "supplier" or input to several practices of the OPF and OPD process areas. There is no clear specification of the minimum information that must be collected. The types of information collected are defined in the model: the organization's common set of measures plus other measures, lessons learned, process improvement proposals, and process artifacts. For measures, the "minimum" is the organization's common set of measures applicable to the processes performed. The assessment team will have to judge whether what is collected, stored, and, more importantly, used is appropriate to the OSSPs, the projects, and the organizational and support groups. Too much can be as big a problem as too little.