Confirmative Evaluation


Confirmative Evaluation

Confirmative evaluation is "a new paradigm for continuous improvement." [50] Twenty years ago Misanchuk introduced confirmative evaluation as a logical next step to formative and summative evaluation of instructional materials or Iearners. [51] Today, with the impact of the quality movement on evaluation, practitioners within the instructional and PT environments are beginning to accept the fact that "Quality control requires continuous evaluation including extending the cycle beyond summative evaluation ." [52] Confirmative evaluation builds on the findings and recommendations generated during formative and summative evaluation. It enables evaluators to confirm the continuing worth of performance improvement packages and performers and helps organizations decide how to manage continuous performance improvement efforts.

Definition and Scope

The following process definition of confirmative evaluation, although written for instructional developers, easily fits the broader context of PT: "Confirmative evaluation is the process of collecting, examining, and interpreting data and information in order to determine the continuing competence of learners or the continuing effectiveness of instructional materials." [53] The definition is easy to translate into "PT-talk" by substituting the word performers for learners and performance improvement package for instructional materials.

Confirmative evaluation "challenges us to jettison linear models and integrate evaluative process throughout every phase of [PT]." [54] For further insight into what confirmative evaluation is all about, it is necessary to examine its purpose and timing.

Purpose

The major purpose of confirmative evaluation is to provide continuous quality control over the life cycle of the performance improvement package. Within the PT context, confirmative evaluation does this by placing a value on the endurance of the performance improvement package and by helping the decisionmakers establish what to do next. [55]

Confirmative evaluation identifies, explains, and confirms the value of the performance improvement intervention over time. "Enduring or long- term effects refer to those changes that can be identified after the passage of time and are directly linked to the performance improvement package." [56] Figure 7-7 below illustrates how the value of a performance intervention is equal to the continuing competence of the performers who participate in the intervention and the continuing effectiveness of the entire performance improvement package including products and processes.

Value of the Intervention

=

Continuing competence of the performers

+

Continuing effectiveness of the performance package


Figure 7-7: EQUATION FOR CONFIRMING THE VALUE OF A PERFORMANCE INTERVENTION

Confirmative evaluation also helps decisionmakers select ways to manage the performers and the performance improvement package over time. Table 7-6 illustrates the six options that are open to the decisionmakers. The table also suggests that even if the level of performance meets performance standards or the performance package is effective, there is always the option to exceed the standards, increase the effectiveness, and delight the customer.

 
Table 7-6: SIX OPTIONS FOR MANAGING THE PERFORMER AND THE PERFORMANCE IMPROVEMENT PACKAGE
click to expand

In addition to confirming the competency of the performers and the effectiveness of the performance improvement package, there are other reasons why PT practitioners or evaluators should conduct confirmative evaluation:

  1. Confirmative evaluation may be used to link the intervention to broader accomplishments that directly affect the organization and to establish actual costs and benefits. [57]

  2. Long-term formative evaluation, or confirmative evaluation, not only supports and strengthens continuous improvement efforts within an organization, but can also "feed back valuable hypotheses to researchers in pursuit of a better understanding of human behavior, performance, and accomplishment," helping to build a scientific base for PT. [58]

When Is Confirmative Evaluation Used?

There are two viewpoints on when to use confirmative evaluation. The first views confirmative evaluation as an extension of formative evaluation, and the second views confirmative evaluation as a separate and distinct form of evaluation that goes beyond formative and summative evaluation.

The first viewpoint stresses that confirmative evaluation should be "an ongoing process designed to 'take a pulse' before, during and after an intervention is implemented. Confirmative evaluation must be woven into the fabric of an intervention...." [59] This viewpoint is in-line with the concept of "long-term formative evaluation," which extends formative evaluation beyond implementation and summative evaluation. "Long-term formative evaluation is conducted after the newly installed intervention has been in effect for some time, and after its novelty has worn off. Ideally, this evaluation should be repeated every six months or so." [60]

This section will focus on the second and more traditional viewpoint that "the major element which distinguishes confirmative evaluation from formative and summative evaluation is the time factor. Confirmative evaluation... takes place after implementation." [61] The heuristic, or rule of thumb, is that confirmative evaluation should take place six months to a year after implementation of the performance improvement package. The following guidelines say that confirmative evaluation should not begin until and unless:

  1. The criteria for the formative and summative evaluation phases have been met.

  2. Detailed assessment information on the participants is available.

  3. Instrumentation is available to assess the areas of possible effect and other intervening factors. [62]

Another way to determine when to conduct confirmative evaluation is to use these three criteria: criticality, complexity, and frequency. [63] For example, if the performers and the organization rate the criticality, complexity, or frequency of a performance as eight or higher on a scale of 1-10, then confirmative evaluation of any intervention to improve that performance should be conducted every six months after implementation. The job aid at the end of this section will provide guidelines for determining when to conduct a confirmative evaluation based on the three criteria.

Conducting a Confirmative Evaluation

Because long-term formative evaluation and confirmative evaluation are so similar in intent, the following guidelines for conducting long-term formative evaluation [64] also apply to the process of confirmative evaluation:

  1. Collect data every six months.

  2. Be as unobtrusive as possible; for example, incorporate data collection into ongoing activities such as performance appraisals or audits .

  3. Focus on the typical effects of the package such as long-term productivity and payoff.

  4. Use a form of expert review to verify that the content of the package is still valid, timely , and aligned with the latest corporate policies.

Methods for conducting confirmative evaluation may include, but certainly are not limited to, the following:

  • Checklists (desired or optimal performance)

  • Interviews (performer, supervisor, coworkers, customers, etc.)

  • Observation (performance tests)

  • Rating scales (quantifies level of actual performance)

  • Assessments (printed tests of knowledge and skill; pre-and post-tests)

  • Review of existing information (before-and-after data, safety reports, production reports , suggestions etc.)

These and other methods may be used separately or together. For example, in the following situation, the performance standards were known and the PT practitioner planned to train observers to use checklists and a rating scale to conduct confirmative evaluation.

One task assigned to the field service managers at automotive dealerships is to appraise leased vehicles that have been turned in at the end of a lease. The managers were not consistent in their appraisals. To minimize the variance between the appraisers, a performance improvement package was implemented. The package included classroom and hands-on training, job aids, and incentives.

In addition to formative evaluation, the evaluation plan called for both summative and confirmative evaluation. Immediately after the initial implementation of the package, and again six months later, evaluators would observe the managers as they appraised a series of vehicles with preset conditions. The evaluators would use a checklist, which included a list of the conditions, the standards, and a rating scale for each performance. The resulting data would be compared and used to determine the continuing competence of the performers and the continuing effectiveness of the package.

Confirmative Evaluation Model

In 1997 Performance Improvement published a model of "confirmative evaluation-a new paradigm for continuous improvement." [65] The model divides confirmative evaluation into four phases: planning, doing, assessing, and improving. The focus of the model is on "continuous improvement, a principle driven by customer expectations." [66] The model, and its companion checklist, are shown in Figures 7-8 and 7-9.

click to expand
Figure 7-8: MOSELEY AND SOLOMON CONFIRMATIVE EVALUATION MODEL
click to expand
Figure 7-9: PRELIMINARY CHECKLIST FOR CONFIRMATIVE EVALUATION

The tasks to be conducted during each phase of the confirmative evaluation model are listed in Table 7-7.

Table 7-7: TASKS TO PERFORM DURING CONFIRMATIVE EVALUATION PHASES

During this phase...

The evaluator will...

  1. Planning

Focus and design the evaluation

  1. Doing

Collect information using multiple information gathering techniques such as questionnaires, interviews, observations, focus groups, work sample analysis, performance analysis, context studies, peer-supervisor-self reports, cost-benefit analysis

  1. Assessing

Gather and interpret data

  1. Improving

Consider the impact, value added, and effectiveness, then recommend whether to continue, improve, or terminate the intervention

Who Should Conduct Confirmative Evaluation?

Confirmative evaluation requires a special perspective and strong analysis skills. One option is to use a team of " unbiased evaluators" [67] to determine whether or not the performers have maintained the level of competence they achieved after the package was implemented and whether or not the package itself still meets the original objectives. The same source also suggests using an outside evaluator teamed with an expert who is familiar with the performance or the environment in which the performers function. The following case study shows how outside evaluators confirmed the effectiveness of a performance intervention package that was implemented on a grand scale.

start sidebar
Case Study: Confirming the Long-term Effects of a Nationwide Reading Program

Situation

In the 1990s there was a movement toward introducing reading and writing skills into the K-12 curriculum at the kindergarten and even preschool level. This is an example of how a confirmative evaluation helped to determine national educational policy.

During the 1973 “74 school year, a number of school districts throughout the U.S. implemented the SWRL/Ginn Beginning Reading Program in their kindergartens. Several years later, Hanson and Siegel wrote a technical report that described how a portion of these students compared with students from the same district who had different kindergarten experiences. [68] The instrument used to compare the students was specifically designed to measure both current reading skills and other potential effects of the kindergarten reading program. The confirmative evaluation was also based on extensive summative evaluation data gathered from a large national sample of schools that implemented the program in their kindergarten classes.

Intervention

Participating school districts offered a formal beginning reading program in kindergarten instead of first grade. The goal was to increase the reading ability of students in the K-12 system by exposing them to beginning reading concepts and skills at an earlier age than traditional school curricula allowed. Specifically, the program sought to increase reading ability, decrease the need for remediation , and foster an ongoing positive attitude toward reading.

Results

The high school students showed a clear and consistent pattern of increased reading competency as a result of receiving formal reading instruction in kindergarten. Those high school students who went through the reading program in kindergarten were superior readers to those who did not take the program. The students who took the program also had higher grades, better attendance patterns, more positive attitudes toward reading, and less need for remediation. The results of this confirmative evaluation were parallel to the results from similar long-term evaluations of early intervention programs in science curricula.

Lessons Learned

The case for using confirmative evaluation to determine policy in the field of education was affirmed by this study and can be extrapolated to other environments such as business and industry. The study provided insights on the many purposes of confirmative evaluation including:

  1. Helps decisionmaker to develop policy that will produce desirable long-term effects, for example, a policy to introduce reading skills into the kindergarten curriculum based on confirmation of increased student performance.

  2. Provides a rational basis for improvement during the life cycle of a program.

  3. Helps resolve any gaps between the original stated goals and the actual outcomes .

  4. Produces cost-benefit information, for example, the need for less remediation may decrease costs within a school district

This case study was written by Joan Conway Dessinger, Ed. D., The Lake Group , based on personal experience and a report by Hanson and Siegl. Used with permission.

end sidebar
 
Job Aid 7-4: DETERMINING WHEN TO CONDUCT A CONFIRMATIVE EVALUATION
start example

Here are some heuristics, or rules of thumb, for determining when to conduct confirmative evaluation based on three performance factors: criticality, complexity, and frequency. Gather responses from as many stakeholders as possible: performers, their supervisors, managers, customers, vendors , suppliers, and organizational decision makers . Then use this job aid to determine when to conduct a confirmative evaluation.

If this performance factor...

Is rated...

Confirm the "package" every...

How critical is the performance to the success of the organization?

  • High

  • Medium

  • Low

  • 6 months

  • 6 “12 months

  • 12 months

How complex is the performance?

  • High

  • Medium

  • Low

  • 6 months

  • 6 “12 months

  • annually

How often is the performance required?

  • Regularly

  • Monthly

  • Annually

  • One time only

  • 12 months

  • 6 months

  • 12 months

  • 6 months or do not confirm

How often is the performance intervention applied?

  • Regularly

  • Monthly

  • Annually

  • One time only

  • 12 months

  • 6 months

  • 12 months

  • 6 months or do not confirm

ISPI 2000 Permission granted for unlimited duplication for noncommercial use.

end example
 

[50] Moseley and Solomon, 1997, p. 12

[51] Misanchuk, 1978, p. 15

[52] Seels and Richey, 1994, p. 59

[53] Hellebrandt and Russell, 1993, p. 22

[54] Moseley and Solomon, 1997, p. 12

[55] Brinkerhoff, 1987

[56] Hanson and Siegel, 1995, pp. 27 “28

[57] Hanson and Siegel 1995

[58] Thiagarajan, 1991, p. 34

[59] Moseley and Solomon, 1997, pp. 12 “13

[60] Thiagarajan, 1991, p. 31

[61] Hellebrandt and Russell, 1993, p. 24

[62] Hanson and Siegel, 1995, p. 28

[63] Hellebrandt and Russell, 1993

[64] Thiagarajan 1991, p. 31

[65] Moseley and Solomon, p. 13

[66] Moseley and Solomon, 1997, p. 13

[67] Hellebrandt and Russell, 1993, p. 27

[68] Hanson and Siegel, 1995




Fundamentals of Performance Technology. A Guide to Improving People, Process, and Performance
Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance
ISBN: 1890289086
EAN: 2147483647
Year: 2004
Pages: 98

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net