Management model and tool
Up one levelSummary of contents:
A management oriented approach to the evaluation of e-learning
Report on the testing of the management evaluation tool
A management oriented approach to the evaluation of e-learning
The management oriented approach was developed by CRED and Pontydysgu from Wales.
Background and introduction
The management-orientated approach to evaluation is meant to serve decision makers. Its rationale is that evaluation information is an essential part of good decision-making and that the evaluator can be most effective by focussing the evaluation products on the needs of managers, policymakers, administrators and practitioners. (Fitzpatrick, Sanders and Worthen 2004)
Although it ignores the needs of other stakeholders, we believe it can be a useful and appropriate approach to the evaluation of e-learning. Managers of VET institutions, owners of SME and human resource professionals in large companies are having to make decisions about the introduction and use of e-learning when e-learning itself is still in a stage of rapid evolution and instability. Major paradigm shifts are taking place in the pedagogical thinking underpinning e-learning, new ideas and policies are emerging on how e-learning should be developed and financed and there are continuing advances in information and communication technologies.
It is in this context that managers are having to make decisions about investing in e-learning and one in which the consequences of making the wrong decision are increasingly costly. Thus
...the decision maker is the audience to whom a management-oriented evaluation is directed and the decision makers concerns, informational needs and criteria for effectiveness guide the direction of the study. (Fitzpatrick, Sanders and Worthen 2004)
We are not advocating a management oriented approach to the evaluation of e-learning as being preferable to any other, rather that it has specific advantages which can be appropriate in particular contexts.
It is particularly suited to evaluators and managers who are most comfortable with a rational and orderly approach
It gives a sharp focus to an evaluation and limits the range of data to that which is relevant to the pending decisions of the managers
It stresses the importance of the utility of the information
Connecting decision-making and evaluation underscores the purpose of evaluation
Focussing an evaluation on the decisions a manager must make prevents the evaluator from pursuing unfruitful lines of enquiry that are not of interest to decision makers or over which they have no decision making control
Instrumental in showing evaluators and managers that they need not wait until a programme has run its course before evaluating it and emphasising that evaluation should begin when ideas for programmes are first discussed
Preferred choice in eyes of most managers and executive bodies – which is hardly surprising as it puts their needs in pole position
Answers a common criticism of evaluation - that it does not provide useful information
| Type of decision to be informed (CIPP) | Type of evaluation | Purpose |
|---|---|---|
| Context evaluation | Planning decisions | Helps define objectives |
| Input evaluation | Structuring decisions | Facilitates programme design |
| process evaluation | Implementing decisions | Allows procedures to be monitored, controllled and refined |
| Product evaluation | Recycling decisions | Enables programme attainments to be judged and informs transfer and dissemination |
(from Fitzpatrick, Sanders and Worthen 2004)
Basic assumptions underpinning management oriented evaluation
Alkin (1991) based much of his practice on the following assumptions
Evaluation is a process of gathering information
The information collected in an evaluation will be used primarily to make decisions about alternative courses of action.
Evaluation information should be presented to the decision-maker in a form that he can use effectively and that is designed to help rather than confuse or mislead him
Different kinds of decisions will need different kinds of evaluation procedures
We were comfortable with taking these as our starting point and what follows is predicated on the above assumptions
Theoretical framework
We have based this tool on Daniel Stufflebeam’s ‘CIPP’ model, which suggests that there are 4 types of evaluation:
Context evaluation
Input evaluation
Process evaluation
Product evaluation
Each one is linked to a particular stage in the lifecycle of a project or programme and is designed to inform particular sorts of management decision.
Context Evaluation determines what needs or problems are to be addressed by the programme, what sort of responses already exist and thus, what the objectives of the programme should be. This can embrace, for example, position auditing, ex-ante evaluation.
Input Evaluation determines what resources are available, what alternative strategies should be considered and what plan seems to have the best potential for meeting the needs. This will inform the programme design. Methods could include scoping studies or feasibility studies.
Process Evaluation determines how well the plan is being implemented, what barriers threaten its success and what revisions might be needed. This will provide the monitoring framework for the programme. Tools such as SWOT analysis may be used.
Product evaluation determines what results were achieved, to what extent was the problem solved or the needs reduced and what should be done with the programme after it has finished. That is, what should be sustained, developed and transferred. This may involve capitalisation, dissemination and valorisation studies being undertaken.
Using the tool
We have based our tool on the sorts of planning, structuring, implementing and recycling decisions that managers responsible for e-learning may have to take. Obviously, particular managers will be faced with particular decisions and this tool can only provide a template that each evaluator will want to adjust to reflect individual circumstances. There will be decisions which, in reality, managers will have to take which do not appear in our schema and there will be decisions that we have included that will not be relevant to all e-learning managers. However, the tool can at least provide a starting point and a systematic way of approaching the evaluation of an e-learning programme.
Another problem is that although we can identify four types of evaluation, the reality is that evaluators may be brought in at any stage of the program life-cycle. So an evaluator may be brought at the beginning of the planning stage in order to gather information that will inform planning decisions. This is often perceived as a consultancy rather than an evaluation role. Conversely, an evaluator may be brought in at the end of the planning phase to review the planning process and generate information which will inform the next stage or which can inform similar planning decisions in the future. These are quite different roles and for that reason the model (which tries to support all of these possibilities), suffers from having to make choices about, for example, syntax and tenses. We have ended up with a compromise, assuming the viewpoint of an evaluator who is retrospectively looking different stages of an e-learning programme in order to inform future decisions. Individuals should adjust the questions if this is not the context and timeframe in which they are operating.
Want to know more?
The tool can be downloaded from this page of the web site. There are three variants of the tool – one for evaluating e-learning development in Small and Medium Enterprises, a second for e-learning in large companies and a third for e-learning development in institutions.
-
A management oriented approach to the evaluation of e-learning
[ Download ]
(mantoolfin.doc
-
422.00 Kb
)
- This file provides details of the background to the development of the management tool. It gives information on the use of the tool plus full details of the tool itself.
-
Report on the testing of the management evaluation tool
[ Download ]
(mantestfin.doc
-
86.50 Kb
)
- This is a report on the testing of the management evaluation tool
