Appendix P: Measurement Plan Outline | Interpreting the CMMI (R): A Process Improvement Approach, Second Edition

Checklist for Evaluating the Effectiveness of the Pilot

Set Up

Was this the first-ever attempt to do a pilot? (Either the first attempt within the organization, or the first attempt planned and administered by this group of people.)
Was this the first attempt to pilot this particular issue/process/ approach?
Was the pilot project of small enough scope that its results could be meaningful for future implementation decisions?
Have personal or political biases ( visions of success or failure of the pilot) been injected into the pilot process or the pilot plan?
Have costs and benefits been addressed in planning?
Have alternative approaches been taken into consideration?
Has planning for re-piloting been addressed?

Design

Was an evaluation plan prepared?
Did the plan contain evaluation and measurement criteria?
Were key factors for evaluation identified?
Were all affected parties identified?
Were plans made to consider and include (as appropriate) the opinions and reactions of the affected parties pertaining to the results of the pilot?
Was provision made for collecting and analyzing cost-benefit information?
Was there an analysis identifying appropriate cost-benefit information?

Conduct

Was the primary focus of the pilot on evaluating the process/issue/ approach addressed in the pilot, or was the motivation for the pilot biased in any way?
Were pilot results documented?
Were lessons learned recorded?
Is associated documentation adequate?
Were feedback mechanisms implemented to foster proper evaluation of the pilot?
Were all the affected parties' constituencies approached for their reactions?
Were these reactions documented and analyzed ?
Was cost-benefit information collected and analyzed?

Evaluation

Was there a documented evaluation phase?
Was there a serious attempt to evaluate the pilot results?
Were the reactions of affected parties collected and analyzed?
Were recommendations resulting from the pilot based on lessons learned and not on personal/political biases?
Was there a formal decision-making process regarding the future use of the pilots in the organization?
Was there a formal decision-making process regarding the future use of the results of this pilot in the organization?
Were costs and benefits compared?
To what extent did the pilot work/not work?
Were cost and benefit data included as part of the decision-making process?
If the pilot did not work, are there any potentially beneficial results that can be drawn from this pilot?
Was there a plan for evaluation, and was it followed?
Were evaluation and measurement criteria identified?
Was a methodology or tracking system for data collection defined?
Were the tasks involved in that methodology carefully defined?
Were evaluation conclusions clearly stated and well supported?
Is sufficient evidence presented to support the conclusions drawn?
How confident are you in all of the results of the pilot?
Do we need to re-pilot?

Implement

Were evaluation results used in a decision to implement?
Were alternatives to the pilot concept explored during planning for implementation?
In the decision making regarding implementation, are costs and benefits addressed?
Is there a discussion in the decision making of potential savings, or of monetary impact of implementation/nonimplementation?
In the decision-making process, were costs and benefits, and other insights, gained during the conduct of the pilot, as well as possible alternatives, addressed?
Was implementation based on the original pilot methodology, or modifications to it?
Were cost-benefit and schedule considerations adjusted due to any such modifications?
Were accurate cost and benefit data kept during implementation and use?
Was a comparison made between estimated versus actual costs, benefits, schedule improvement, and quality?
What modifications to the product or process must be made to make this work?
What lessons learned can we draw?
Were the pilot tests true pilot efforts? (A true pilot will follow the phases of the pilot life cycle, especially concerning Set Up and Planning.)
Are the pilots concerned only with technical feasibility, or do they also examine the cost and schedule effectiveness of the proposed and alternate solutions? What, if any, cost and benefit measures were used?
Did pilot monitoring and control procedures follow the organization's standards and processes?
Did formal, documented evaluation processes exist for pilots? If so, did they yield valid results?
For pilots that led to system or process implementation, were the claimed/ projected benefits actually realized?