Tool for the Evaluation of the Effectiveness of E-learning Programs in Small and Medium sized Enterprises (SMEs)

The Tool for the Evaluation of the Effectiveness of E-learning Programs in Small- and Medium-sized Enterprises has been developed by the Institute for Future Studies, Innsbruck.

Introduction

The evaluation tool has been designed to provide an easy-to-use instrument to carry out a retrospective evaluation of an e-learning programme. This retrospective analysis should enable enterprises to detect the weaknesses and strengths of their learning programme with regard to its organisational, pedagogic and technological implications.

The tool is based on Daniel Stufflebeam’s CIPP Model, a model which is often used for evaluation in education. In order to gain a more specific view of e-learning measures in small and medium-sized enterprises (SME), however, the model had to be adapted, and besides other minor changes, three dimensions were introduced for each phase of the evaluation: organisation, pedagogy and technology.

These three dimensions reflect central fields of concern in connection with the introduction of e-learning in SME. Through this division a multi-facetted view of the introductory stages of the new programme is gained, which enables the evaluator to track down both potential problem areas and structures supportive to the programme in a more detailed way. Consequently, control mechanisms and measures to revise and adapt certain areas of concern may be applied in a more specific manner.
 
The design of the tool is based on a set of questions incorporating central fields of concern in the introduction of e-learning measures in SMEs. The sets of questions developed for the tool are intended as a guideline for the evaluation of relevant aspects of the respective e-learning programme. The evaluation questions were not intended to be too precise or prescriptive, in order to enable flexibility in the application of the tool. Consequently, the questions have to be adapted to the specific circumstances characterizing an enterprise’s e-learning program. This adaptation may involve a reformulation or even the omission of sets of questions.

The main aim of the evaluation is to help managers of SMEs detect strengths and weaknesses of the e-learning program they are currently running, and thus providing them with particular information on organisational, pedagogic and technological implications of the training measure.

The structure of the CIPP Model, on which the tool is based, allows for a well-structured evaluation of planning-, structuring-, implementing- and recycling decisions. Therefore, the tool may be applied to track down possible weaknesses at a certain point of the program-cycle and assist (programme) managers in the introduction of potential adaptive measures. Above all, the aim of the in-depth analysis of the various stages of the programme cycle is the improvement of the training programme as a whole, and at the same time providing a precise tool for the detection of possible weaknesses and strengths within the cycle.

Furthermore, the amount to which the e-learning program was successful or not is identified through the retrospective evaluative perspective of the tool. This especially applies to the issue of knowledge acquisition and the appropriateness of learning measures. Therefore, another main point of the evaluation is whether the e-learning measures applied by the enterprise assisted the acquisition of knowledge and fulfilled the training needs defined at the beginning of the training programme. If the training results do not correspond with the training needs outlined at the start of the programme, possible factors which led to the unsatisfactory results can be traced back to certain phases within the programme cycle. Through this analysis both learner and tutor satisfaction can be measured.

In addition to the points mentioned above, evaluation findings should provide enterprises with additional information on a successful implementation of e-learning products as well as providing help for future acquisitions of e-learning products. This also implies the detection of an e-learning strategy that best suits the purpose of the respective enterprise.

Moreover, the evaluation aims at supporting the enterprise in finding ways of making training programmes more sustainable. For this purpose not only the training budget is scrutinized, but also other organisational, pedagogic and technological matters are checked. Eventually, the question is raised of whether the programme itself or certain aspects within the programme could be transferred to other areas within the enterprise.

Finding out more

More details of the SME model and tool are in the documents which can be downloaded from this page. If you would like to test the tool yourself contact Friedrich Schuermann - office@friedrich-scheuermann.net.