Level 5

Checklist for Planning and Conducting Pilots

Set Up:

1. Define the problem that is to be addressed.

2. Generate alternatives, if possible.

3. If alternatives exist, examine alternative solutions to the problem for technical feasibility and cost/benefit.

4. Choose at least one alternative solution for piloting.

5. Identify the key points to be measured.

6. Define the methods and techniques that will be used to draw conclusions about pilot results (e.g., committee of senior managers, EPG, survey of the pilot project members , definition of success/failure, supporting data to be collected), including technical, operational, and financial capability.

Design:

7. Document the processes and procedures to be used on the pilot.

8. Define the pilot evaluation criteria. Include quality factors.

9. Define the methods, processes, roles, personnel, timing, report formats, and data collection activities needed to perform the evaluation.

10. Identify all affected parties whose input and ideas are beneficial to the pilot. Do not include "squeaky wheels" at this point.

11. Define ways of gathering and tracking cost and benefit data.

12. Derive a schedule and plan for the pilot. Include milestones for the pilot and associated deliverables.

Conduct:

13. Conduct the pilot according to the plan generated during the Design phase:

    • Execute the processes.

    • Contact all affected parties.

    • Collect the necessary data.

    • Achieve the milestones and deliverables.

14. If the design could not be followed, log and explain all deviations.

15. Record all issues that occurred during the conduct of the pilot, important thoughts about those issues, and how the issues were resolved.

16. Save all data and intermediate and final results for evaluation and lessons learned.

Evaluate:

17. Conduct the evaluation according to the Plan.

18. Use the evaluation methods and techniques:

    • Investigate results.

    • Analyze data.

    • Achieve the milestones and deliverables.

19. If the evaluation plan could not be followed, log and explain all deviations.

20. Record all issues encountered during the evaluation, important thoughts about those issues, and how the issue was resolved.

21. Save all data and intermediate and final results for lessons learned.

22. Draw conclusions about the workability and effectiveness of the approach that was the subject of the pilot, based on pilot results.

23. Conduct a cost-benefit analysis as part of the final decision to implement the results of this pilot.

24. Gather all of the lessons learned, refine them, and document them for future use and applicability on other pilots.

25. Make recommendations regarding future use.

26. Analyze the pilot itself, in regard to how it was conducted and for ways of improving future pilots.

Implement:

27. Base implementation decisions on pilot evaluation results and recommendations, and not on personal or political biases.

28. Base implementation decisions on any ongoing alternatives analyses, comparing pilot findings with information from updated alternatives.

29. Base implementation decisions on any other pilots that may occur in the organization.

30. Continue to gather data to determine the effectiveness of the pilot results when implemented in the organization (costs-benefits, schedule improvement, quality improvement, etc.).

31. Continue to collect, document, and analyze lessons learned.