A Framework for the evaluation of e-learning

From a baseline of practice of attempting to evaluate many e-learning programmes, one of the biggest problems has proved to be handling the number of variables which potentially impact on the effectiveness of the programme and deciding what constitutes dependent, independent and irrelevant variables in a given situation.
 
Literature reviews and the study of existing evaluation practice, suggests that many evaluation tools and schema tend to disregard – consciously or otherwise many of these variables. Much of existing practice is overly focused on the technology – and on learner reaction to the use of technology. Socio-economic factors such as class or gender are seldom considered and even learning environment variables such as the subject environment are all too often ignored.

Not only does this result in limitations in the data available on the use of ICT in learning but the limited recognition of the different variables can distort analysis of the weaknesses (and strengths) in current e-learning provision.

The Evaluation of e-learning project has developed a more comprehensive framework. Over several e-learning evaluation projects, five major clusters of variables have emerged; individual learner variables, environmental variables, technology variables contextual variables and pedagogic variables.  Each of these can be disaggregated into more precise groups and further disaggregated until individual variables can be identified and isolated.

Of course we recognise that no single evaluation model or tool, much less evaluation study, can address every variable. But, we believe in approaching and designing any evaluation it is important to be conscious of what factors are being disregarded or edited out of the framework.

Individual learner variables include

• physical characteristics (e.g age, sex, physical abilities)
• learning history, (negative / positive experience, level of attainment, duration, recency etc)
• learner attitude (positive / negative)
• learner motivation (high / low)
• familiarity with the technology

Learning environment variables include

• the immediate (physical) learning environment
• the organisational or institutional environment
• the subject environment

Contextual variables include

• socio-economic factors (e.g. class, gender,)
• the political context (e.g. who is funding /paying for the e-learning and for what reason ?)
• cultural background (e.g. how highly is learning / e-learning valued ?)
• geographic location (e.g. country, language, urban/rural)

Technology variables  include

• hardware
• software,
• connectivity,
• the media
• mode of delivery,

Pedagogic variables include

• Level and nature of learner support systems
• accessibility issues.
• Methodologies
• Flexibility
• Learner autonomy
• Selection and recruitment
• Assessment and examination
• Accreditation and certification

 

How might this framework be used ?

Firstly the framework can be used to develop a robust classification system with clearly identified levels of aggregation, (which themselves may be context determined.) for mapping and coding existing work into the effectiveness, efficiency and economy of e-learning irrespective of whether this is an evaluation or an independent research study. Methodologies are cross-referenced against the variables being studied and major areas of omission can be identified that in turn will suggest a future research agenda.
Secondly we are using the clusters of variables can be sued for proposing and testing hypotheses. Any one cluster can act as the dependent variable; the other 4 then operate as independent variables. For example, at the micro level, part of the Eval project has tested the hypothesis that the effectiveness of different e-learning pedagogies will depend on particular individual learning histories. Another survey explored whether the effectiveness of particular technologies depends on gender. At a macro level we are also interested in whether the presence (or absence) of some individual variables or clusters of variables are more significant than others in determining the effectiveness of e-learning and, if so, can they be weighted in some way? Is the profile of the learner more significant than the nature of the learning environment? Is the effectiveness of the technological solution outweighed or enhanced by particular environmental variables? Which is more important – getting the software right or the learner support right? Can we use statistical techniques such as factor analysis to see which variables ‘cluster’ together and impact on each other?

We were not able to test every variable in the limited time and resources available to us through he project. However, the research we were able to undertake proved the value of the framework as a tool for research and confirmed the validity of the framework design.
Thirdly, we have found it a useful framework for evaluating and researching the effectiveness of specific e-learning projects and programmes. The evaluation of e-learning, and research into the evaluation of e-learning, has been dominated by descriptive ethnographic studies, rather than interpretation and analyses and there is a predominance of ethno-methodological approaches, in particular, heavily contextualised case studies. The relatively small number of empirical studies has focussed on a limited number of variables. The best of these have controlled for variables other than those under study; the worst have simply discounted them. As the databank of research results is built up, particularly as the different variables are `weighted’, it becomes easier to identify the irrelevant variables and allow for the impact of others. It also allows predictions to be made which can short circuit the search for an appropriate evaluation methodology.