ࡱ> 02/ bjbj"$l 2$ ~%%:zLr| P0VPPAbstract ECER 2004 EVAL3-project Symposium on Models and tool for the evaluation of eLearning Part 1. Literature study (Yvonne de Jong and Wilfried Admiraal, Utrecht University) Within the framework of the European Leonardo-programme, a project has been set up (EVAL3) to review the existing research and literature into the evaluation of e-learning in European VET. It will generate a taxonomy or framework for classifying, storing and accessing the existing research and will map these evaluation reports and outcomes onto the framework. In addition, new tools, instruments and models will be developed. A web-based literature survey revealed that the existing research is very limited and can be grouped under seven main headings (see Hughes & Attwell, 2003). Case studies of specific e- training programmes For the most part these are descriptive rather than analytic or predictive, predominantly American, mainly located in a Higher Education rather than vocational training environment and focussed on the virtual classroom model. Comparisons with traditional learning There are some (but surprisingly few) systematic studies that compare e-learning effectiveness with traditional learning and which are empirically robust. Those that exist are mainly small-scale studies, often using a matched pairs design and are frequently of very specific instances of e-learning in which the e-learning methodologies are idiosyncratic and the conclusions are non-generalisable. Tools and instruments for evaluation of e-learning There is an abundance of literature detailing tools for the evaluation of e-learning. However, these are mainly divided into two types. Firstly there are many on-line data gathering instruments for assessing, typically, the user interface characteristics of software (e.g. student perception questionnaires) or secondly, there are devices to record and analyse useage by duration and frequency of log-in, pages accessed, user profile etc. Many of these are sophisticated in their design and ingenuity but lack guidance on interpretation and analysis. Return on Investment (ROI) reports There are surprisingly few ROI reports, considering the huge investments into e-learning at all levels. The majority of those that exist draw mainly from industry based examples and are written from an HRD perspective. The conclusion is inevitably that the investment was cost-effective and represented value-for-money but often the savings are defined in efficiency rather than effectiveness with no long-term impact analysis that takes account of unintended outcomes and consequences. Many ROI type evaluation reports appear to be justifying investment rather than evaluating it and more geared to an audience of shareholders rather than researchers. Benchmarking models There have been several attempts to generate sets of criteria for quality assuring e-learning. However, these tend to be skewed towards proposing quality standards for e-learning systems and software which often disregard key variables in the wider learning environment or are based on criteria associated with evaluating traditional learning processes (and which disregard the technology) or criteria associated with measuring learner achievement through traditional pedagogies. Product evaluation By far the greatest number of `hits on evaluation of e-learning are reports describing (and extolling the virtues of) particular education software. The vast majority of these reports are commissioned or published by the software developers. This is not to question the usefulness of these reports or necessarily to doubt their validity but evaluation of `de-contextualised software is not an acceptable substitute for the rigorous evaluation of e-learning systems. Performance evaluation In the USA, the term `performance evaluation is used for what would, in European terms, be called student assessment. Whilst it is true that an examination of student performance is a powerful indicator of the effectiveness of e-learning, it is by no means the only one. Moreover, a survey of reports on performance evaluation in the context of e-learning were mainly concerned with on-line tools and instruments for examining knowledge-based learner performance and could therefore be categorised under that heading. In this first part of the symposium, we will go into the international literature on models, tools and instruments of evaluation of eLearning. Most of the literature and empirical work concentrate on benchmarking models and learning-oriented evaluation. This is particularly true for the evaluation of eLearning in programmes in higher (profesisonal) education. The results will be presented from a systematic literature study, including various sources (Current contents, Eric, Psychinfo, Socio Abstracts, Ibiss, Internet searches), and (combination of) relevant terms such as evaluation, eLearning, ICT, technology, learning, knowledge management, VET, professional education, workplace, adn work-based learning. This search resulted in some 100 relevant papers and articles. "/^``6CJ 5CJ\CJ CJmHsH!"_``0< % & I  & Fd d d  1h. A!"#$% i>@> StandaardCJ_HaJmH sH tH4`4 Kop 1$d @& 6CJ]\@\ Kop 2$ & Fd<@&!5OJQJ\]^JaJmHsH`@` Kop 3$ & Fd<@&%56CJOJQJ\^JaJmHsHDA@D Standaardalinea-lettertype6B`6 Platte tekstd CJ$!"_``0<%&I 00000000 000 000 000 000 000 000 0000!hivlos/P:\SCHOVO\ECER2004\Abstract ECER 2004 EVAL3.docivlos;D:\temp\AutoHerstel-versie van Abstract ECER 2004 EVAL3.asdivlos;D:\temp\AutoHerstel-versie van Abstract ECER 2004 EVAL3.asd{#3  Db7 Qb M(iV;hh^h`o(.P8^`Po(3.^`o(3...x^`xo(....  ^`o( .....  X@ ^ `Xo( ......  ^ `o(....... 8x^`8o(........ `H^``o(.........h^`.hP^`P..h0^0`...h(xp^(`x.... h @ ^ ` ..... h X ^ `X ...... hx^`....... h8^`8........ hH`^H``.........hh^h`o(.P8^`Po(2..^`o(...x^`xo(....  ^`o( .....  X@ ^ `Xo( ......  ^ `o(....... 8x^`8o(........ `H^``o(.........h^`CJOJQJo(qh ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( D D Q Q{#(ij K@tP@UnknownGz Times New Roman5Symbol3& z Arial;Wingdings?5 z Courier New"1ӣݣӣd"!0  2q Abstract ECER 2004 EVAL3-projectivlosivlosOh+'0 $0 L X d p|!Abstract ECER 2004 EVAL3-projectrosbstivlosctvlovlo Normal.dotEivlos.d3loMicrosoft Word 9.0E@2~@b{@b{@vd՜.+,0 hp|  uu" 2 !Abstract ECER 2004 EVAL3-project Titel  !"#$%&()*+,-.1Root Entry F 31TablePWordDocument"$SummaryInformation(DocumentSummaryInformation8'CompObjjObjectPool    FMicrosoft Word-document MSWordDocWord.Document.89q