A testing problem

In Glenford Myers’ book The Art of Software Testing, he points to a simple program containing 20 lines of code, including a loop and three IF statements. He calculated that there were 100 trillion execution paths, which would take a billion years to test! Given that a typical e-learning application, with its requirement for sophisticated interaction, will be many times more complex, you would be forgiven for concluding that quality control in e-learning is a hopeless task.

Not so, according to Mark Aberdour, Manager of EpiCentre, the UK’s largest e-learning testing service. “In the hands of well-trained testing professionals, it is possible to deliver a robust product that delivers to specification. Yet, despite overwhelming evidence regarding the cost-effectiveness of testing, a large proportion of the e-learning industry remains either sceptical or unaware of the need to test rigorously.”

The effects of inadequately tested e-learning courses can be devastating. According to a landmark study by IBM and Rockwell, a bug found post-release costs on average 100 times more to fix than a bug found early on in development. Says Aberdour: “In e-learning terms that could mean help-desk support, learners unable to complete training, negotiating rework with developers, an additional re-work cycle and another roll-out of the product to the organisation. For a high profile project, this could easily run to millions. In spite of this, testing continues to be done late in the product cycle, corners are cut to save time and the work is often carried out by junior, untrained staff.”

According to Aberdour, a content developer should allow something between 10 and 15% of project time for testing. Even when this is allowed, project delays often mean that testing time is cut back. To avoid this, Aberdour recommends that the test team have an early involvement in the development process, as bugs found early on are cheaper to fix.

Quality control is not just a process of eliminating bugs. E-learning projects have many facets, requiring different quality control measures. First and foremost there is a need for ‘functional testing’, to ensure that the product conforms to requirements, typically expressed in a technical specification or design document. Functional testing includes a check on functionality (to make sure it does what it says on the tin), on content (to remove spelling and grammatical errors) and on compatibility (to ensure it works on all the required platforms, operating systems and browsers).

Functional testing requires a mix of skills, resources and methods. Testers not only have to check the product against its spec, they also need to look for the unexpected. Says Aberdour: “We allow a couple of days for destructive testing, when our testers look for all those things that the designers didn’t expect learners to do. This always helps us to identify important defects that could cause a product to fall over in the field.”

To test for compatibility, EpicCentre have an enormous range of different hardware and software at their disposal. Although the software industry would rather like it that everyone used the latest version of Internet Explorer, running under Windows XP, with a full range of the latest plug-ins, on a Pentium 4 with a broadband connection, the reality is sadly rather different. Reaching a wide audience means supporting a range of platforms and that isn’t so easy, to develop or to test. EpicCentre track all the problems that they find in their bug database, an off-the-shelf product called Problem Tracker, which helps them to monitor who is accountable for resolving a problem and where it is in terms of resolution.

Interest is increasing in what EpicCentre call ‘non-functional testing’, which covers a myriad of important issues, including accessibility (for the disabled), usability, conformance to industry standards such as SCORM, and security. Each of these is an important consideration for an organisation publishing or distributing e-learning and each requires a different approach from the test team.

Quality standards and processes

Total Quality Management

The first major quality assurance movement, based principally on the work of W. Edwards Deming and applied to great effect in Japanese industry. TQM stressed the need for a ‘right first time’ approach by all participants in a work process, the use of ‘quality circles’ to ensure continuous improvement, and a way of looking at quality objectively in terms of ‘fitness for purpose’.

ISO 9001

ISO 9000 is a group of standards and guidelines for quality management and quality assurance. ISO 9001-2000, the most recent standard, applies to manufacturing and service firms and public agencies. To conform to ISO 9001, organisations must adopt work processes that conform to the standards and be certified as compliant by an awarding body.

Carnegie Mellon Software Engineering Institute: Capability Maturity Model

A wide-ranging software quality model employed by over 60 major organisations worldwide. Covers areas such as the management of technological change, the reduction of errors and process change management. Emphasises that ‘what can’t be measured can’t be managed’ and ‘what can’t be measured and managed, can’t be improved’.

Six Sigma

A customer-based approach that acknowledges that defects are expensive and that fewer defects mean lower costs and improved customer loyalty. Six Sigma aims for a defect level of less than 3.4 per million opportunities. The underlying process consists of five steps: define, measure, analyse, improve and control, or DMAIC.

Institute of IT Training: e-Learning Standards

Comprehensive standards for the design of e-learning materials, endorsed by major industry bodies such as the e-Learning Network.

The World Wide Web Consortium (W3C) Web Content Accessibility Guidelines

Guidelines for the development of web content that can be accessed by the disabled.

SCORM (Sharable Courseware Objects Reference Model)

Technical standards to ensure the interoperability of e-learning content with e-learning platforms, such as Virtual Learning Environments and Learning Management Systems.

Institute of IT Training: Website Usability Standards

A set of 200 guidelines to help website designers create more usable websites.