The Policy model and tool

The policy model and tool has been developed by CRED and Pontydysgu, both based in Wales.

Background

In developing a Framework for the evaluation of e-learning. we recognised the breadth of variables impacting on the quality of e-learning. Five groups of variables were identified:

Individual learner variables include

• physical characteristics (e.g age, sex, physical abilities)

• learning history, (negative / positive experience, level of attainment, duration, recency etc)

• learner attitude (positive / negative)

• learner motivation (high / low)

• familiarity with the technology

Learning environment variables include

• the immediate (physical) learning environment

• the organisational or institutional environment

• the subject environment

Contextual variables include

• socio-economic factors (e.g. class, gender,)

• the political context (e.g. who is funding /paying for the e-learning and for what reason ?)

• cultural background (e.g. how highly is learning / e-learning valued ?)

• geographic location (e.g. country, language, urban/rural)

Technology variables include

• hardware

• software,

• connectivity,

• the media

• mode of delivery,

Pedagogic variables include

• Level and nature of learner support systems

• accessibility issues.

• Methodologies

• Flexibility

• Learner autonomy

• Selection and recruitment

• Assessment and examination

• Accreditation and certification

We were concerned that the unconscious disregarding of many of these variables diminished the quality of many models and tools for evaluating e-learning. This is even more so when it comes to evaluating e-learning policy, which by its very nature, may impact on many different aspects of teaching and learning.

Therefore, in seeking to develop a model and tool for evaluating e-learning policy, we wished to find a means of measuring all the different variables, or at least a means of expressing the value of the different variables, even if all could not be studied in any single evaluation report.

Developing the model and tool

The first stage was to take each of the variables – which we called impact variables and to derive the differentiating factors for that variable.

An example of this is given below for the individual variables.

Secondly we recognised that each variable would impact on each other. Thus it was possible to develop differentiating variables for individual variables when combined with – for example technology variables – and to identify what should be key factors in any e-learning policy development. This is illustrated in the table below.

Individual x technology variables

Hardware

The e-learning policy should

• ensure learners have access to computers and other hardware of a quality and a quantity sufficient to meet their e-learning needs

• ensure learners have technical support for hardware systems

• promote mechanisms to improve stability and robustness of hardware systems to minimise disruption to e-learning

Software

The e-learning policy should

• promote the development of different types of e-learning software to support different learning processes.

• support measures to improve the quality of the ICT learning environment

• ensure there is effective communication between e-learners and between e-learners and teachers /tutors.

• promote mechanisms to improve stability and robustness of software to minimise disruption to e-learning.

Bandwidth and connectivity

The e-learning policy should

• ensure sufficient bandwidth is available to support the type of on-line learning applications being used

• support different options in providing connectivity and bandwidth

• promote economies of scale and increased user access through partnership solutions

Types and combinations of media and modes of delivery

The e-learning policy should

• ensure e-learners have opportunities for face to face experiences in conjunction with their e-learning.

• Support e-learning which is not course based

• Be flexible enough to support learning programmes that combine different e-learning opportunities, different media and / or are blended with non-ICT media and materials.

Using the tool

In theory it is possible to look at every one of the different variables produced by the model. But in reality this is somewhat unlikely – there are simply too many. However, any policy development is likely to be viewed from one of a number of different perspectives. Different stakeholders will have different viewpoints on what a policy should contain and what it should achieve.

Each of the groups of variables and respective list of evaluation issues will have a different associated stakeholder (or group of stakeholders) perspective:

Learners

• Individual learners

• Collectivities of learners

• Social partners

• Funding bodies

Context

• Government and government agencies (Agencies affected by the outputs of e-learning or provide inputs)

• Policy bodies both educational and economic

• ‘public’

Environmental (The institutions that create the learning environment)

• institutional managers

• learning system designers

• content developers

Technology

• standards bodies

• software designers

• infrastructure providers

• technical staff inc decision makers and technicians

Pedagogy

• individual teachers

• other education professionals e.g. advisors, administrators, researchers, counsellors

• learners

• communities of learners

• examining and validating bodies

• quality assurance bodies

• ‘guardians’ of subject knowledge

• community of practice of teaching (both formal and informal)

• gatekeepers.

This approach has the strength of recognising multiple viewpoints and perspectives in evaluation of policy. An evaluator does not have to be a member of the different stakeholder group they represent in undertaking an evaluation. It is enough that the different perspectives are recognised in selecting the evaluation questions to be asked.

Even when the variables are narrowed to those of a particular perspective it is recognised that the range and quantity of evaluation issues and questions may still be too large for many evaluation initiatives and it may prove necessary to select from the range of questions on offer. But, at least now this selection is a conscious one, rather than evaluation perspectives and variables having been unconsciously disregarded.

Testing the model and tool

Two tests were made of the tool – one in Finland and the other in Wales. In both instances the testers found the initial model a little hard to grasp. However, both agreed on the value of the approach in providing a comprehensive yardstick against which to measure a policy.

It was not easy to find policies with which to test the tool and it emerged that policy development in this field is often haphazard and often documentation is sketchy. One recommendation from the testing is that the tool may best be used as part of a process of policy formation, rather than of post formation evaluation.

Want to find out more?

The model and tool can be downloaded from this page of the web site.