Evaluation Models Show the Way

Models provide blueprints for conducting evaluation by illustrating when, what, or how to evaluate. Below are some models that are useful guides for the PT practitioner or evaluator .

Geis and Smith Model

Geis and Smith use the basic Instructional System Design (ISD) Model ADDIE (Analyze, Design, Develop, Implement, Evaluate) to illustrate when to evaluate PT interventions. ^[17]

Geis and Smith's model (Figure 7-3) emphasizes a proactive, continuous improvement approach to evaluating performance intervention processes and products.

Figure 7-3: GEIS AND SMITH EVALUATION MODEL

Dessinger-Moseley 360 ° Evaluation Model

The Dessinger-Moseley 360 ° Evaluation Model ^[18] (Figure 7-4) uses spiralling concentric circles to represent "the proactive and iterative nature of evaluation." ^[19] In fact, the authors may literally (with the help of computer technology) set the concentric circles spinning to reinforce that evaluation should be continuous. "Each phase uses the prior phase for validation (and) the entire evaluation process is evaluated (meta evaluation) in terms of changing needs and lessons learned." ^[20]

Figure 7-4: DESSINGER-MOSELEY 360 ° EVALUATION MODEL

Table 7-3 compares the HPT Model and the Dessinger-Moseley Model.

Table 7-3: COMPARISON OF THE HPT MODEL AND THE DESSINGER-MOSELEY 360 ° EVALUATION MODEL

HPT Model	Dessinger-Moseley Model
Performance Analysis Cause Analysis	Evaluate analysis process and products
Intervention Selection Intervention Design	Evaluate design/development (or selection) process and products
Intervention Implementation Change	Evaluate immediate user reaction Evaluate immediate user accomplishment Evaluate on-the-job transfer Evaluate organizational impact
Formative Evaluation Summative Evaluation Confirmative Evaluation	Evaluate all evaluation processes and products ” Meta Evaluation

Kirkpatrick Evaluation Model

The most well-known model of training evaluation is the Kirkpatrick Model. ^[21] Kirkpatrick originally developed the model in 1959 to clarify the concept that the potential for evaluating training programs exists at four levels (see Figure 7-5).

Level 1 ”Immediate Reaction

Level 2 ”Immediate Learning

Level 3 ”On-the-Job Behavior

Level 4 ”Organizational Results

Figure 7-5: KIRKPATRICK'S EVALUATION MODEL

Although this model specifically addresses training programs, it is widely used by both Instructional Systems Design (ISD) and performance technology(PT) practitioners because it provides "not only a common language, but also a common tool that lets us compare results." ^[22]

Kaufman-Keller-Watkins Model

Recognizing the limits of the Kirkpatrick model, Kaufman, Keller, and Watkins (Figure 7-6) offered an expanded version designed to fit the PT environment:

We suggest that these levels (Kirkpatrick Level 1-4) are incomplete, however, in terms of assessing performance and consequences, and as such they have encouraged many to focus narrowly on evaluating training.... Including interventions other than training into the general evaluation design will require some modification to the four levels, while incorporating existing aspects of program evaluation.... We propose that the principles and processes of training evaluation be expanded to consider all interventions associated with strategic and tactical planning, performance improvement, organizational development, customer satisfaction/total quality and societal contributions. ^[23]

Level 1	Input and Reaction ”the availability and quality of human, financial, and physical resources (input) and the perceived acceptability and efficiency of method and processes (reaction)
Level 2	Acquisition ”individual and small- group mastery and competence
Level 3	Successful Application ”individual and small-group utilization within the organization
Level 4	Organizational Results ”organizational contributions and payoff
Level 5	Societal Consequences ”societal and client responsiveness, contributions, and payoffs

Figure 7-6: ADAPTATION OF THE KAUFMAN-KELLER-WATKINS MODEL

Brinkerhoff Six-stage Model

Another evaluation model that is less well-known but also applicable to evaluating performance interventions is the Six-stage Model by Brinkerhoff. ^[24] Brinkerhoff looks at evaluation as a cycle and says that his model "responds to the decisions necessary for programs (or interventions) to proceed productively and defensibly ... enabling and facilitating quality efforts." In PT terms, the Six-stage Model evaluates the following:

1. Needs and goals ” the value and importance of the problems or opportunities that trigger the intervention

2. Design ” the practicality, soundness, and responsiveness to the needs and goals

3. Operation ” the "goodness" of the installation and implementation of the performance intervention in relation to the needs, goals, and design

4. Learning ” the level of the user's knowledge, skills, and attitudes when he or she first uses the intervention on the job

5. Usage and endurance ” how well the intervention achieves the intended results over time

6. Payoff ” the return on investment from the implementation of the intervention

^[17] Geis and Smith, 1992

^[18] Moseley and Dessinger, 1998

^[19] Moseley and Dessinger, 1998, p. 247

^[20] Dessinger, 1997

^[21] Kirkpatrick, 1994

^[22] Carliner, 1997

^[23] Kaufman, Keller, and Watkins, 1996, p. 9

^[24] Brinkerhoff, 1987, p. 26