Background and introduction

The management-orientated approach to evaluation is meant to serve decision makers. Its rationale is that evaluation information is an essential part of good decision-making and that the evaluator can be most effective by focussing the evaluation products on the needs of managers, policymakers, administrators and practitioners. (Fitzpatrick, Sanders and Worthen 2004)

Although it ignores the needs of other stakeholders, we believe it can be a useful and appropriate approach to the evaluation of e-learning. Managers of VET institutions, owners of SME and human resource professionals in large companies are having to make decisions about the introduction and use of e-learning when e-learning itself is still in a stage of rapid evolution and instability. Major paradigm shifts are taking place in the pedagogical thinking underpinning e-learning, new ideas and policies are emerging on how e-learning should be developed and financed and there are continuing advances in information and communication technologies.

It is in this context that managers are having to make decisions about investing in e-learning and one in which the consequences of making the wrong decision are increasingly costly. Thus

We are not advocating a management oriented approach to the evaluation of e-learning as being preferable to any other, rather that it has specific advantages which can be appropriate in particular contexts.

• It is particularly suited to evaluators and managers who are most comfortable with a rational and orderly approach
• It gives a sharp focus to an evaluation and limits the range of data to that which is relevant to the pending decisions of the managers
• It stresses the importance of the utility of the information
• Connecting decision-making and evaluation underscores the purpose of evaluation
• Focussing an evaluation on the decisions a manager must make prevents the evaluator from pursuing unfruitful lines of enquiry that are not of interest to decision makers or over which they have no decision making control
• Instrumental in showing evaluators and managers that they need not wait until a programme has run its course before evaluating it and emphasising that evaluation should begin when ideas for programmes are first discussed
• Preferred choice in eyes of most managers and executive bodies – which is hardly surprising as it puts their needs in pole position
• Answers a common criticism of evaluation - that it does not provide useful information

Basic assumptions underpinning management oriented evaluation

Alkin (1991) based much of his practice on the following assumptions

• Evaluation is a process of gathering information
• The information collected in an evaluation will be used primarily to make decisions about alternative courses of action.
• Evaluation information should be presented to the decision-maker in a form that he can use effectively and that is designed to help rather than confuse or mislead him
• Different kinds of decisions will need different kinds of evaluation procedures

We were comfortable with taking these as our starting point and what follows is predicated on the above assumptions

Theoretical framework

We have based this tool on Daniel Stufflebeam’s ‘CIPP’ model, which suggests that there are 4 types of evaluation:

• Context evaluation
• Input evaluation
• Process evaluation
• Product evaluation

Each one is linked to a particular stage in the lifecycle of a project or programme and is designed to inform particular sorts of management decision

Context Evaluation determines what needs or problems are to be addressed by the programme, what sort of responses already exist and thus, what the objectives of the programme should be. This can embrace, for example, position auditing, ex-ante evaluation.

Input Evaluation determines what resources are available, what alternative strategies should be considered and what plan seems to have the best potential for meeting the needs. This will inform the programme design. Methods could include scoping studies or feasibility studies.

Process Evaluation determines how well the plan is being implemented, what barriers threaten its success and what revisions might be needed. This will provide the monitoring framework for the programme. Tools such as SWOT analysis may be used.

Product evaluation determines what results were achieved, to what extent was the problem solved or the needs reduced and what should be done with the programme after it has finished. That is, what should be sustained, developed and transferred. This may involve capitalisation, dissemination and valorisation studies being undertaken.

Using the tool

We have based our tool on the sorts of planning, structuring, implementing and recycling decisions that managers responsible for e-learning may have to take. Obviously, particular managers will be faced with particular decisions and this tool can only provide a template that each evaluator will want to adjust to reflect individual circumstances. There will be decisions which, in reality, managers will have to take which do not appear in our schema and there will be decisions that we have included that will not be relevant to all e-learning managers. However, the tool can at least provide a starting point and a systematic way of approaching the evaluation of an e-learning programme.

Another problem is that although we can identify four types of evaluation, the reality is that evaluators may be brought in at any stage of the program life-cycle. So an evaluator may be brought at the beginning of the planning stage in order to gather information that will inform planning decisions. This is often perceived as a consultancy rather than an evaluation role. Conversely, an evaluator may be brought in at the end of the planning phase to review the planning process and generate information which will inform the next stage or which can inform similar planning decisions in the future. These are quite different roles and for that reason the model (which tries to support all of these possibilities), suffers from having to make choices about, for example, syntax and tenses. We have ended up with a compromise, assuming the viewpoint of an evaluator who is retrospectively looking different stages of an e-learning programme in order to inform future decisions. Individuals should adjust the questions if this is not the context and timeframe in which they are operating

Before you start

1. Identify the stage in the process at which you, as an evaluator, are being asked to intervene. That is, are you undertaking a context evaluation to inform planning decisions or an input evaluation to inform structuring decisions or a process evaluation to inform implementing decisions or a product evaluation to inform recycling decisions.
2. Is the evaluation summative (retrospective) with the purpose of informing decisions in the future or is it formative – starting at the beginning of the process and informing current decisions?
3. Identify the key decision makers responsible for the programme. These could be managers of the organisation who are buying the training (e.g. mangers of company buying e-learning for workers) or managers of the organisation providing the training. In a large company, both providers and customers could be part of the same organisation.
4. Work closely with the decision-maker(s) to identify the decisions they have to make and a time frame for those decisions. This could be short, medium or long term.
5. Check out your understanding of the task. The following mental checklist may help evaluators clarify their thoughts before they start using the tool.

What do they want out of the evaluation?

• Why do they want me there?
• Who do I report to?
• What do they want to know?
• Are they going to let me find out?
• Do they want advice on what decisions to make or how to make the decisions?
• Is their decision making open to change?
• Who do the decisions affect and who has a say?
• What are the goals of the decisions being made?
• Do they want these goals to be achieved?
• How will the evaluation help them do that?
• From whose perspective / range of perspectives am I working?
• With what purpose/aim?
• Are they trying to resolve problems as they go along?
• Are they concerned with improving decision making in the future?
• What types of thing will I be measuring?
• How will I measure things?
• What measures will I be using?
• Who do I need to talk to?
• Who do they care about?
• Why?
• What is required as an evaluation output and what is desired?
• How would they like the evaluation reported? In what format?
• How am I going to record my findings?
• Do they want strict advice or guidelines for the future?
• Do they want to change things or is it an ‘ok’ stamp?
• What do they not want to know about?
• What do they want to hear and what don’t they want to hear?
• Do they want a qualitative statement or a descriptive analysis?

The first column represents the sort of decisions that managers have to make. The second column includes the sorts of questions that an evaluator may want to ask to generate information that will inform the decisions. These are not intended as an interview proforma but more as a basis for dialogue with the clients and areas the evaluator should be investigating. The third column suggests additional tools the evaluator may want to use or recommend.