Project Summary

E-val 3 is a two year Leonardo da Vinci funded project. The full project name is 'Models and Instruments for the evaluation of e-learning and ICT supported learning'. The project is co-ordinated by Pontydysgu and the partners are the the University of Utrecht, Netherlands; Interactive Studio, Sweden; CRED, Wales; the University of Tampere, Finland; the Institut für Zukunftsforschung, Austria

What

Year 1 of the project will review the existing research into the evaluation of e-learning, collate and cross reference the results and extrapolate those which have particular implications for the evaluation of e-learning in European VET. It will review and report on present deficiencies in the research, establish major research questions and propose a research agenda. The eval-3 project will generate a new taxonomy and framework for classifying, storing and accessing the existing research and will map existing research and e-learning evaluation reports and outcomes onto the framework. It will also produce an XML standards generated template for reporting on research on the evaluation of e-learning and establish an XML standards database of this research which will complement as the existing CEDRA database of VET research and the E-VAL database of evaluation research and use the same platform. Year 2 of the project will start with a critique of existing models of evaluation and establish (or otherwise) their fitness for purpose and use for the evaluation of e-learning in VET in Europe.

The project will also propose new models for the evaluation of e-learning for European VET which take account of different learner groups, different learning environments, different socio-cultural contexts and different technologies. These models will be field tested and backed up with small scale empirical field research to test their validity. Year 2 will also generate a set of tools and instruments to support the evaluation of e-learning in European VET targeted at particular market deficiencies. These will depend on individual partner interests but should include e-learning applications in informal or community environments, in work based learning, school based VET and, in particular, a set of macro level evaluation tools for policy makers and planners in VET

Why

The development of e-learning products and the provision of e-learning opportunities is the most rapidly expanding area of education and training. It is also the area that attracts the most research funding. However, what is known about the effectiveness of these innovative approaches to training has been limited by the shortage of scientifically credible evaluation. If this investment is to be maximised, it is imperative that we generate robust models for the systematic evaluation of e-learning and produce tools which are flexible in use but consistent in results. Although recent attention has increased e-learning evaluation, the current research base for evaluating e-learning is inadequate…Due to the initial cost of implementing e-learning programs, it is important to conduct evaluation studies. American Society for Training and Development 2001 The existing literature on e-learning evaluation is limited to comparatively few organisational and social contexts (predominantly HE), has been restricted to certain countries, (predominantly America) and focuses on the ‘virtual class room’ model.

What is lacking is a theoretical basis and a coherent research framework. There is little systematic research into broad based issues and concepts, or the generation of transferable models and processes of evaluating e-learning or into the design of tools for analysing, rather than collecting, data. Furthermore, there are few papers written which collate the results of the existing research and classify it in an accessible way. Nor is there substantial evidence of work that extrapolates and tests generalisable principles arising from the case studies and surveys or which comments on the implications or application of these results in a European VET arena.

How - methodology

One of the biggest problems in the evaluation of e-learning is handling the number of variables which potentially impact on the effectiveness of the programme and deciding what constitutes dependent, independent and irrelevant variables in a given situation.

The project team have already suggested four major clusters of variables viz individual learner variables, environmental variables, technology variables and contextual variables. Each of these can be further disaggregated into more precise groups. For example, individual learner variables will include physical characteristics (e.g age, gender), learning history, learner attitude and motivation etc. Variables relating to the learning environment will include both the immediate physical and organisational environment, the learner support systems and accessibility issues.

Contextual variables will include socio-economic factors, the political context and cultural background. Technology variables will include both hardware and software, connectivity, the media and mode of delivery, the implicit pedagogy connected with each and so on. Each of these groups can in turn be further disaggregated until individual variables can be identified and isolated. The project will initially undertake a survey of practitioners to test out the validity of the above taxonomy, which will also be informed by a search of existing literature and use the results to build a robust research classification system with clearly identified levels of aggregation, which themselves may be context determined. The partners will also undertake empirical studies to establish the relative importance of each of these variables in a particular situation and ‘weight’ them accordingly. The studies will also determine, using factor analysis, which variables ‘cluster’ together and impact on each other.

Research into the evaluation of e-learning has been dominated by descriptive ethnographic studies, rather that interpretation and analyses, and there is a predominance of ethnomethodological approaches, in particular, heavily contextualised case studies. The relatively small number of empirical studies have focussed on a limited number of variables. The best of these have controlled for variables other than those under study; the worst have simply discounted them. The project will attempt to draw together the key findings from all these studies and map them onto the framework, cross referencing methodologies against the variables being studied and identifying major areas of ommission which will suggest a future research agenda.