Seven levels of learning evaluation

Now that there has been some exploration on the intention and conduct of evaluation, it is time to give some structure to how you may go about specifically measuring the impact and quality of your learning and innovation in your business.

Let us explore a model of seven levels of learning evaluation. Starting with Donald Kirkpatrick who identified the first four levels of evaluation and, in more recent years , Jack Phillips who helped move the discussion to a higher plane (that is, the fifth level) of return on investment (ROI). From there two more levels will be suggested. The sixth is sustainability and the seventh is shared purpose. These help lift the exploration of results to include better examination of business governance and the triple bottom line.

This explanation begins with Kirpatrick s work of levels 1 to 4, then proceeds to level 7.

Level 1: Reaction

This level measures individual reaction to a course or experience. This form of evaluation is typically the most common used in business, probably because it is perceived as the easiest to do. This form of evaluation is normally seen as an evaluation sheet at the end of a training course. Here we can discover whether a person had a good time and found it meaningful. This lowest form of evaluation can raise many questions rather than answers. For example, just because someone had a wonderful time does not necessarily mean that they learnt something valuable . The reaction method can also be used to evaluate on-the-job coaching and e-learning.

Level 2: Learning

Here attention shifts to what knowledge skills and attitudes have actually changed or have been acquired during an experience. Changes in learning are commonly determined with some combination of pre- and post-assessment as well as direct observation and testing. Sometimes this evaluation will resolve whether new knowledge or learning has been put to use but generally the discussion is quite shallow .

Level 3: Behaviour

Level 3 measures real change as a result of the initiative being undertaken. This normally involves direct observation of people s performance on the job. For this level of evaluation to be successful it requires assessment which is accurate and unbiased . The assessors are normally the managers, coaches or mentors of the people actually performing the work. To do this well, training and good education on assessment is vital . A skilled assessor will note not only behavioural change, but also which factors might inhibit doing the job in the first place, for example, a lack of resources, unclear policy or some other hindering factor.

Level 4: Business results

This identifies how learning and knowledge is delivered to the business, and typically involves assessment against measures such as productivity, customer retention, sales and profitability. Depending on the nature of your business, your interest in knowledge and in innovation, this measure will vary. In undertaking this evaluation care must be taken to separate those factors that have nothing to do with the initiative, for example, external pressures such as changed market conditions.

Level 5: Return on investment

Return on investment shows the cost versus the benefits of an initiative. In undertaking such a study you will discover the linkage between learning and improved profitability, efficiency and effectiveness. For example, did the dollar value of the produced benefits exceed the cost of hosting and setting up the initiative? Some creativity will be required to quantify the hard-to-measure competencies and capabilities. In most cases some measurement can be placed on the data you are receiving; the key is to be clear about the assumptions you are using. There are a number of ratios and metrics used. A simple measure recommended by Jack Phillips is

 Total benefit (in $) x 100 ROI% = ---------------------------        Total programme cost 

Level 6: Sustainability

At this sixth level of evaluation we shift our attention to broader and more outward focused questions. For example, will the knowledge or learning help the business pass the test of time? Here the issue is not just the economics of learning and performance, but whether the capabilities and competencies being learnt are actually helping a business prepare for the future. We need to pay close attention to the context and environment of change and be receptive to new views of what excellence means. Businesses which invest in their people and systems, typically keep a close watch on their environment and will have a higher chance of measuring success here.

‚

ROI% = Total benefit (in $) 100

Total programme cost

Level 7: Sharing the benefit

The highest and most altruistic level of evaluation is asking whether your business know-how is adding value and helping others, whether for your suppliers, customers, partners or society as a whole. As has been discussed previously, on the triple bottom line business is increasingly expected to be accountable for its actions and be a good corporate citizen not just for our current generation but for future ones as well. So, we must be prepared to have independent assessment as well as contribute to the quality of the life of others.

As a postscript to the seven levels of learning evaluation, the key to ensuring lasting success is to move up the hierarchy. If you are able to raise questions and stimulate conversations on the higher levels, you will most likely advance over time. Here are some examples of what people have said about actual knowledge and innovation initiatives:

• ˜Customers now feel they are not lost in the system.

• ˜It has enabled peer support and best practice sharing.

• ˜We now view problems from a common perspective.

• ˜We are learning from both our successes and failures.

• ˜We have a much better response to customer needs.

• ˜There is much better communication and openness.