Evaluating e-learning

The development of e-learning products and the provision of e-learning opportunities is one of the most rapidly expanding areas of education and training. Whether this is through an intranet, the internet, multimedia, interactive TV or computer based training, the growth of e-learning is accelerating. However, what is known about these innovative approaches to training has been limited by the shortage of scientifically credible evaluation. Is e-learning effective? In what contexts?  For what groups of learners? How do different learners respond? Are there marked differences between different ICT platforms? Does the socio-cultural environment make a difference? Considering the costs of implementing ICT based training, is there a positive return on investment? What are the perceptions of VET professionals? What problems has it created for them?

E-learning is also one of the areas that attracts the most research and development funding. If this investment is to be maximised, it is imperative that we generate robust models for the systematic evaluation of e-learning and produce tools which are flexible in use but consistent in results.

“Although recent attention has increased e-learning evaluation, the current research base for evaluating e-learning is inadequate…Due to the initial cost of implementing e-learning programs, it is important to conduct evaluation studies.” (American Society for Training and Development 2001).

However, whilst some have been desperately seeking answers to the question `What works and what doesn’t work?’ and looking for ways of improving the quality of e-learning, the response by a large sector of the community of e-learning developers and practitioners has been a growing preoccupation with software and platforms.  There has been only limited attention to pedagogy and learning. The development of models and tools for the evaluation of e-learning can help in improving the quality of e-learning and in informing and shaping future.

This section of the web site is given over to a description of the different models and tools developed through the project. We do not suggest this is a comprehensive selection – but it does illustrate the different models and tools which can be deployed. Central to our project’s findings are that evaluating e-learning is no different than evaluating any other form of learning – but that there are many variables and that the models and tools must take account of the different aims of the evaluations and contexts in which e-learning is taking place.

The following models and tools have been developed through the project and are described in more detail on the linked pages.

Models and tools for the evaluation of eLearning in higher vocational education

The learner-benchmarking tool developed for the evaluation of eLearning in higher vocational education is a consumer tool in order to evaluate the use of Virtual Learning Environments. It is based on an on-line questionnaire including items on the appraisal of the VLE used by the teachers in their courses.

The Learner Evaluation tool is focused on the guidance or scaffolding teachers provide within the VLE and is based on constructivist principles.

A management oriented approach to the evaluation of e-learning

The management oriented approach to the evaluation of e-learning is a model and tool designed to provide formative feedback for decision making in developing and implementing an e-learning programme. It is based on the CIPP model. There are three variants of the tool for

• Educational institutions;
• small and medium enterprises;
• Large companies

Models and tools for evaluating e-learning policy

The models and tools for evaluating e-learning policy are based on the framework for the evaluation of e-learning. The five different groups of variables are further broken down to provide differentiating factors or criteria against which a policy can be evaluated. The tool is designed for uses in policy evaluation at any level – national, regional or institutional.

Tool for the Evaluation of the Effectiveness of E-learning Programs in Small and Medium sized Enterprises (SMEs)

The model and tool for the evaluation of the effectiveness of e-Learning programs in SMEs has been designed to provide an easy-to-use instrument to carry out a retrospective evaluation of an e-learning program. This retrospective analysis should enable enterprises to detect weaknesses and strengths of their learning program with regard to its organisational, pedagogic and technological implications.

Evalact

Evalact is an electronic tool, designed to allow facilitate the evaluation of individual learning. It allows the creation of on-line questionnaires and provides rich feedback and statistical evidence for evaluators. It also allows the comparison of different evaluation instances and provides graphical interfaces for assisting in the analysis of evaluation data.

The  SPEAK Model and Tool

SPEAK is a model and tool for the self evaluation of learning in a group and community context. It is designed both to facilitate group discussion and self evaluation and to provide accumulative data for programme evaluation. The SPEAK system has been designed to incorporate a number of inter-related functions within a logical framework.

• Strategic Planning: producing information to assist both staff and management to make informed decisions
• Evaluation: ongoing internal review; cross project periodic assessments; with a facility for external inputs.
• Knowledge: networking based on sharing information between institutes, countries, regions and themes.

The SPEAK model has been developed as an electronic tool distributed by a CD ROM. Data can be aggregated and queries through a server application.