Chapter 7: Evaluation


Introduction to Evaluation

Authors and practitioners alike seem to agree that to evaluate is to place a value on or judge the worth of a person, place, thing, or event. In the context of PT, evaluation is a way to "compare results with intentions and delve into the usefulness of methods and resources so that we may move toward the required results." [1]

The International Society for Performance Improvement (ISPI) has long supported "the integral role of evaluation in PT and in the ongoing functioning of any organization." [2] According to ISPI's HPT Model shown below, evaluation is "one of the basic components of PT," [3] along with performance analysis, cause analysis, intervention selection and design, implementation, and change management.

The authors of Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance have enhanced the HPT Model by adding the three types of evaluation formative , summative , and confirmative ”and the meta-evaluation process to the evaluation component. Enhancing the evaluation component (see Table 7-1) broadens the scope of the component and increases the value of the entire HPT Model.

 
Table 7-1: THE FULL SCOPE OF EVALUATION
click to expand
click to expand
Figure 7-1: HPT MODEL ” EVALUATION PHASE

Definition and Scope

There are three types of evaluation ”formative, summative, and confirmative ”plus a process called "meta evaluation," which literally evaluates the evaluation. Taken collectively, formative, summative, confirmative, and meta evaluation:

  • Cover the life cycle of the performance intervention from analysis through implementation and evaluation.

  • Guide the decision-making processes involved in why, what, when, and how to evaluate.

Table 7-1 illustrates the scope of evaluation within the HPT Model.

Formative Evaluation

Formative evaluation is diagnostic and is "used to shape or mold an ongoing process...to provide information for improvement...." [4] Formative evaluation is developmental and continuous; it begins during the analysis stage and continues through the selection and design of interventions and sometimes into early implementation.

Summative Evaluation

Summative evaluation focuses on the effectiveness of a performance intervention after it is implemented. Effectiveness is measured by immediate reaction and personal acquisition of knowledge and skills. Summative evaluation takes place during implementation and change management.

Confirmative Evaluation

Confirmative evaluation identifies and explains long- term or enduring effects. [5] It is "the process of collecting, examining and interpreting data and information in order to determine the continuing competence of learners or the continuing effectiveness of instructional materials." [6] Confirmative evaluation builds on and goes beyond formative and summative to place a value on knowledge or skills transfer to the job, organizational impact, and return on investment.

Meta Evaluation

Meta evaluation is a process for "assuring and checking the quality of evaluations." [7] Through meta evaluation the PT practitioner or evaluator validates the formative, summative, and confirmative evaluations and acquires valuable insight into evaluation processes and products. The best way to distinguish among the three types of evaluation and the meta evaluation process is to look at why and when each type of evaluation and meta evaluation should occur during the life cycle of the HPT Model. Table 7-2 provides an overview.

Table 7-2: OVERVIEW OF EVALUATION ”TYPE, PURPOSE, TIMING

Type/Process of Evaluation

Why? (Purpose)

When? (Timing)

Formative

Improve performance intervention package

During performance analysis, cause analysis, and selection or design of interventions

Summative

Determine immediate competence of user and effectiveness of package

During implementation and change management

Confirmative

Determine continuing competence of user and effectiveness of package

6 “12 months after implementation

Meta

Evaluate formative, summative, and confirmative processes to provide insight to evaluator

After confirmative evaluation

Purpose

In broad terms, "the purpose of evaluation is to illuminate and improve the organization," [8] More specifically , the purpose of evaluation is to generate information that will accomplish two outcomes :

  • Help the organization to value or judge the results of a performance

  • Trigger or support a decision regarding the performance, the performer, or, ultimately, the organization itself.

Traditionally, quantitative and qualitative evaluation is used to prove or disprove the value of a performance intervention. When the HPT Model was first introduced in 1992, "Evaluation of Results" was the last step in the model. Marc Rosenberg, one of the authors of the HPT Model, summarized the evaluation component of the Model as follows : "After applying or implementing the PT solution, it's important to monitor it to determine its effect on performance improvement and on the organization." [9]

On the other hand, "Most evaluators would agree that the purpose of evaluation is to affect decision making." [10] Feedback from evaluation can determine whether to begin, maintain, change, or end the performance ”or the performer. To provide feedback that will trigger or support decisionmaking, the PT practitioner or evaluator may focus the evaluation on one or more of the following goals:

  • Confirm that a particular procedure or treatment is being carried out as prescribed.

  • Indicate the effectiveness of some treatment or intervention in order to decide whether to continue, expand, or eliminate the treatment.

  • Audit the current state of affairs.

  • Determine whether the cost of an activity is justified by its effects. [11]

Earlier, this chapter indicated that the type of evaluation helps to determine the purpose of the evaluation. However, there is also a whole matrix of decisions (see Figure 7-2) that determines the purpose of evaluation, and the purpose of evaluation in turn determines when, what, and how to evaluate. [12]

click to expand
Figure 7-2: MATRIX OF EVALUATION DECISIONS

When the purpose of the evaluation is explicit (clear, specific, and detailed), true (real and undistorted), and determined in advance, the resulting when, what, and how will unfold more smoothly.

When to Evaluate

PT practitioners should not think of evaluation as an afterthought or a one-time event. Ideally, "evaluation may occur at any time and with any frequency. It depends on the purpose of the evaluation." [13] Below are two supporting viewpoints from the literature.

  • "During analysis one often carries out the activities earlier defined as evaluative: clarifying decisions to be made, collecting information, and feeding it into the decision process. A good part of the needs assessment of front-end analysis in general is evaluation." [14]

  • "Establish a framework for accountability during the selection and implementation phases and ensure the personal involvement of key decision makers in choosing bottom-line measures to demonstrate the value of their efforts." [15]

Role of the PT Practitioner/Evaluator

Before completing this introduction to the evaluation component of the HPT Model, it is necessary to take a brief look at the role of the PT practitioner or evaluator. The PT practitioner or evaluator is responsible for performing or monitoring five major tasks , [16] whether the evaluation is formative, summative, confirmative, or meta. The tasks include:

  • Setting the boundaries (or focus) by establishing the purpose, goals, objectives, and scope of the evaluation.

  • Selecting the appropriate method (s) based on the purpose and the context (political environment, available resources, time, and cost).

  • Collecting all the information that is feasible , available, and relevant.

  • Analyzing the data using sound qualitative or quantitative methods.

  • Reporting the findings after first determining who needs to know, what they need to know, how best to inform them, and how often to keep them informed.

The remaining sections in this chapter will help the PT practitioner or evaluator to plan and conduct a formative, summative, confirmative, and meta evaluation within the framework of the HPT Model.

Job Aid 7-1: TYPES OF EVALUATION: ADDRESSING THE ISSUES
start example

Directions: The columns are labeled with the four types of evaluation; the rows are labeled to describe issues that must be addressed when planning an effective evaluation. Fill in each cell before the evaluation begins and revise as the evaluation progresses.

 

Formative

Summative

Confirmative

Meta

Primary Audience

       

Primary Emphasis in Data Collection

       

Primary Role of PT Practitioner

       

Primary Role of Evaluator

       

Typical Methodology

       

Frequency of Data Collection

       

Primary Reporting Mechanisms

       

Reporting Frequency

       

Emphasis in Reporting

       

Requirements for Credibility

       

ISPI 2000 Permission granted for unlimited duplication for noncommercial use.

end example
 

[1] Kaufman, Keller, and Watkins, 1997, p. 9

[2] Geis and Smith, 1992, p. 130

[3] Geis and Smith, 1992, p. 131

[4] Geis and Smith, 1992, p. 134

[5] Hanson and Siegel, 1995, p. 27

[6] Hellebrandt and Russell, 1993

[7] Madaus, Scriven, and Stufflebeam, 1987, p. 16

[8] Geis and Smith, 1992, p. 133

[9] Rosenberg, 1996b, p. 9

[10] Geis and Smith, 1992, p. 133

[11] Geis and Smith, 1992, pp. 133 “134

[12] Geis and Smith, 1992, p. 134

[13] Geis and Smith, 1992, p. 138

[14] Geis and Smith, 1992, p. 139

[15] Rothwell, 1996b, p. 283

[16] Herman, Scriven, and Stufflebeam, 1987




Fundamentals of Performance Technology. A Guide to Improving People, Process, and Performance
Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance
ISBN: 1890289086
EAN: 2147483647
Year: 2004
Pages: 98

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net