Real-world testing

Probably the UK’s largest e-learning development project is at learndirect, with more than 40 developers contributing to the product catalogue. Colin Buckley is Operations Manager for the Learning Team: “Before e-learning, technology-based training products were relatively self-contained. They were designed for a particular platform and worked in isolation from other systems. Now, it is impossible to really see how a product performs until it goes live, when it is running within the learner support environment and additional services, such as synchronous and asynchronous tutor support, are being provided.”

Learndirect is implementing a three-stage testing process to ensure that what it delivers meets expectations. Buckley: “At the very beginning of any contract we work with the developer to define very thoroughly what the build will contain and the technical specification that it must conform to. This is easier said that done, because all 40-odd developers seem to read the specification in different ways! At beta stage we then perform a comprehensive functional test, across many combinations of browser and platform. Finally, we upload the software to the real-world system, but not live, for a four-week period. In this time, our hubs and learning centres provide us with valuable feedback on how the product is performing. On top of that, we perform a final bug fix three months after going live. Only then can we be satisfied that the job is done.”

Of course, none of these tests tell you whether the learner is actually learning anything. Encouragingly, learndirect are addressing this issue with a new evaluation project. Says Buckley: “Using software from University College, Northampton, we will be asking selected learners to complete an online questionnaire, to provide us with feedback on pedagogical issues related to the courses they have completed.”

Another content developer that is concerned about the quality of the learning experience is Line Communications Group. Says CEO, Piers Lea: “The developer has an important responsibility to ensure that content is instructionally sound when it comes to screen. There is still a shortage of skill in this area and many customers do not fully appreciate its importance.”

The same can be said for usability testing. Says Lea: “We do usability testing early on at the prototype stage, using members of the target audience. This has been hugely useful in bringing out practical issues relating to how things are labelled, the navigational system and the look and feel. Typically we will leave learners alone with the software and video them, following this up with some formal questions.”

Testing for usability and ‘learnability’ does not come cheap and, in the end, customers have to assess their priorities. It is often said that, given the choice of quality, price or speed of delivery, you can pick any two of the three, but not the lot. Buyers of e-learning content development would be advised to ensure that quality is one of the two that they pick.