The models and tools for evaluation of eLearning in higher vocational education were developed by the Centre for ICT in education, IVLOS, Utrecht University.

Introduction
In the Netherlands, the use of virtual leaning environments (VLEs) in higher education is an issue. It is the most common practice for e-learning in higher education: VLEs are used in almost all courses and programmes across the entire curriculum of all higher educational institutions. Most universities choose Blackboard as the main or only VLE; others use WebCT. Utrecht University has agreed on the option of two VLEs: Blackboard and WebCT, at least until 2007. In order to ground the policy making in this respect, the use of the VLEs has to be evaluated.

Within the framework of the EU project EVAL3, the project partner (IVLOS) developed three models and tested and evaluated two tools.

The first model refers to evaluation from a perspective of the learning process to be triggered by the use of the particular VLE. The second model includes an evaluation of the teachers’ pedagogy in the VLE affecting students’ learning process. The third model is a consumer model including an evaluation in terms of the teachers’ (users) satisfaction with (elements of) the VLE.

Models for evaluation of virtual leaning environments

Model on learning process in virtual learning environments

We are not ‘value-free’ in choosing what kind of aspects of the students’ learning process we evaluate. As designers of learning environment (in higher education) we want to promote the so-called new learning. These innovative ideas have been developed last decades from the more traditional teaching (guided learning) into experiential and action learning (see, Simons, Van der Linden, & Duffy, 2000).

From guided learning to experiential and action learning

In guided learning a trainer or teacher takes all the relevant decisions and the learner can and should follow him or her. He decides about the goals of learning, the learning strategies, the way to measure learning outcomes, and he takes cae of feedback, judgment and rewards. Success factors for guided learning are, for example:

• Taking differences in interests, prior knowledge and abilities into account,
• Good commitment to learning goals through good communication about it,
• Good communication about learning strategies, and
• Timing and quality of rewards and judgment systems.
• Guided learning is like organizing a collective traveling journey in which the guide is an expert who knows the way and who plans the trip.
• In experiential learning, it is not so much a teacher or a predetermined goal that controls the learning. Rather circumstances, personal motivation, other people, innovations, discoveries, experiments, etc determine what and how one learns. Learning is a side effect of the activities one undertakes. Success factors of experiential learning are, for example:
• Interests, knowledge and action-plans of participants are put central;
• The experience itself is the goal. There are no explicit learning goals;
• Each learner can have his / her own tempo;
• Team learning from and with each other is important, and
• Reward and judgment systems are tuned to discoveries and innovations.
• Experiential learning is like a trekking journey in which a group of people undertakes a trip without planning and organizing at forehand.

In action learning there is a much more active and explicit role for learners and learning goals than in experiential learning. Learning is central and not a side-effect, but the learners themselves determine the goals of learning according to needs arising in their actions (at work or elsewhere). It is self-organized and self-planned. Success factors for action learning are, for example:

• Control of learning by learners;
• Opportunities to reflect on learning goals and learning strategies;
• Self-responsibility for their own learning, and
• Opportunity for self-testing.

Action learning is exploring like pioneers who explore new land. It is the need to find a suitable surrounding to start a new life that guides them. There is a sense of urgency that determines the route and destination in a certain perspective.

Learning from a (social) constructive perspective

Within the framework of the recent developments of experiential and action learning, there are many theoretical as well as practical perspectives on learning and how to support the learning process. (Social-)Constructivism is one of the most cited, and misused terms to incorporate innovate ideas about teaching and learning. Jonassen, Peck and Wilson (1999) describe principles educational designers or teachers use in the design of learning environments from a constructivist perspective. These principles are in line with the recent ideas Utrecht University has about teaching and learning in her university. These ideas are formulated in the framework of the harmonizing European educational system, which has been started some years ago as a results of the Bologna agreement. Jonassen and colleagues also relate the constructivist principles and the use of ICT in teaching and learning. They argue that ICT supported learning is only useful (effective and efficient) is learning is active, constructive, reflective, intentional, authentic (contextual and complex), conversational and interactive.

Active learning means that learners are actively manipulating their learning environment and observing the effects of what they have done. So, learners are responsible for thee results of their learning. Learning meaningful imply actions, but actions are not enough. Learners have reflect on their actions and their observations. These reflections could or should lead to the integration of new experiences and ideas with existing knowledge or should at least leads to insight into what the learner has to learn (constructive learning). It is this combination of active and constructive learning what makes learning meaningful. Learning is not a results of only practicing; learners also have to elaborate their knowledge and skills and create or construct new insights. The authenticity of the learning environment not only leads to a better understanding of cases or principles, but also results in a better transformation of learning outcomes to other cases and contexts. To make a learning environment authentic, it should include complex and open tasks, as well as simple ones. Like in the ‘real’ world or job-related practice, people work together and interact in order to learn, and solve problems. Cooperation between learners (both collaboration and conversation) is seen as important as a goal of learning as well as a mean of learning other content. Especially in a formal educational programme like the university curriculum, learning has to be intentional (although unintentional learning is welcomed also). It is important that learners know what their learning goals are and how they will be striving for them.

Model of the teachers’ pedagogy in virtual learning environments

Table 1. Moderator roles and functions in computer conferences (after Paulsen, 1995)

Moderator role                                                Emphasis of different functions

	Organisational	Social	Intelectual
Goal setter	high
Discriminator			high
Host	high
Pace setter	high		high
Explainer			high
Entertainer		high
Lecturer		low	high
Tutor	high		high
Facilitator	high	high	high
Mediator	high	high
Mentor	low		high
Provocateur			high
Observer	low	low	low
Participant	low

The model of the pedagogy used in VLEs is based on the extensive literature teachers’ moderating in computer-mediated communication (see, e.g., Admiraal, Lockhorst, Wubbels, Korthagen, & Veen, 1998) and on that of Paulsen (1995) in particular. Based on literature on computer conferencing, Paulsen describes three essential functions of computer conferencing moderators: the organisational function; the social function; and the intellectual function. The organisational function refers to structuring the conferences, including setting the agenda (the objectives of the discussion, the timetable and procedural rules). As in most conferences, managing the computer conferences is considered a sine qua non of a varied and extensive communication. Another essential moderator function is creating a friendly, social environment for learning, including sending welcome messages at the beginning and encouraging participation throughout, as well as providing lots of feedback on students’ input and using a friendly tone. The most important function of a moderator is that of the educational facilitator (designated as intellectual function by Paulsen): a moderator should focus discussions on crucial points, ask questions and stimulate students to expand and build on comments. Combining these three functions characterises various moderator roles, each with organisational, social and intellectual elements (see Table 1).

Consumer model for teachers in virtual learning environments

In higher education, a much-used model for the evaluation of educational programmes is the consumer model, based on information of the end-users of a particular programme (mostly students). As the VLEs in university teaching has been used by teachers, they will be the source of information for the evaluation within this model. Although there is always a risk of getting only happy sheets from teachers looking back on their own teaching, this model provides a starting point for further analysis of the functionalities and aspects of VLEs. Consumer-oriented evaluation relies heavily on criteria referenced evaluation techniques such as benchmarking.

Methods for developing and testing the tools

We developed and tested two tools to evaluate virtual learning environments in Utrecht University. The first one is a tool to evaluate the learning and teaching process as it has been visualised in the use of a virtual learning environment in higher vocational education, more specific in its teacher education programmes. In this we focuses on what kind of (student) learning has been triggered and how this was done. The second tool is a benchmark tool focusing on the evaluation of virtual learning environments (VLEs) by the teachers (the consumers).

Tool 1: Evaluation of the learning and teaching process

In this tool, we start from the ideas form Jonassen, Peck and Wilson (1999). They describe several principles educational designers or teachers need to use when designing learning environments from a (socio-)constructivist perspective. They also relate these principles to the use of ICT in teaching and learning and conclude that ICT supported learning is only useful (effective and efficient) when learning is active, constructive, reflective, intentional, authentic (contextual and complex), conversational and interactive.

We used a mix of a grounded theory approach and theory-driven approach, working with sensitizing concepts from the theoretical work of Jonassen et al. formulating and reformulating these concepts. Thus, the seven principles mentioned above are the starting point for our design and redesign of the tool.

The first phase in the process was to analyse two courses from the post-graduated teacher education programme that were developed and presented in a VLE (WebCT Vista). This resulted in the clustering of some concepts and a first description of categories. The remaining four concepts are: (1) active learning, (2) constructive, critical thinking and reflective learning, (3) authentic learning and (4) collaborative and conversational learning. An example of a description of the authentic learning concept is: the teacher offers that what has to be learned, for instance, knowledge and competences, in a meaningful, real-life context or in a relevant professional environment. Examples of this context are the daily teaching practice or a case-based and/or problem-based learning environment.

In the second phase of the research, this perspective on the learning process has been combined with three functions of the teachers’ moderating in the VLE, as distinguished by Paulsen: organisational, social and intellectual. Combining these three moderater functions with the four principles of learning, resulted in a matrix with 12 cells. This is a ‘ high inference’ tool, meaning that a lot of interpretation of the evaluator is needed in order to come to a reliable and valid assessment of e-learning as can be observed in the VLEs.

In the third phase of developing and testing the tool, the instructions for the observers using the matrix have been reformulated and standardized in terms of evaluating VLEs in higher vocational education.

Tool 2. On-line benchmark questionnaire

Our second tool is a on-line questionnaire including items on the appraisal of the VLE used by the teachers in their courses. The questionnaire will be administered with all teachers of the Utrecht University (some 6,000) using a VLE (WebCT Vista or Blackboard) in their courses. The items refer to the possibilities of the VLEs, the usability, and the frequency teachers use the VLE and parts of it in their courses. All departments use the same questionnaire and have some additional items which can be different for each department. The questionnaire will be administered within the VLE (see for the questionnaire, http://www2.ivlos.uu.nl/ictexpertisecentrum/index.html; click on Services/European projects/EVAL3/Questionnaire evaluating VLEs). The questionnaire includes six sections with items with mostly pre-structured answering options on a 5-point Likert type scale:

1. Use of functionalities of the VLE;
2. Teaching in the VLE;
3. Appraisal of the VLE;
4. Policy aspects of the use of the VLE;
5. Additional open questions, and
6. Department specific items.

References

Admiraal, W. F., Lockhorst, D., Wubbels, T., Korthagen, F. A. J., & Veen, W. (1998). Computer-mediated communication environments in teacher education: computer conferencing and the supervision of student teachers. Learning Environments Research,1, 59-74.

Jonassen, D.H., & Peck, K.L., & Wilson, B.G. (1999). Learning with technology. A constructivist perspective. Upper Saddle
River, NJ: Prenctice-Hall.

Paulsen, M. F. (1995). Moderating educatinal computer conferences. In Z.L. Berge & M.P. Collins (Eds.),
Computer-mediated communications and the online classroom (Vol IV pp. 81-103) Cresskill, NJ: Hampton Press.

Simons, R.J., Van der Linden, J., & Duffy, T. (2000). New learning: three ways to learn in a new balance. In R.J. Simons, J. Van der Linden, & T. Duffy (Eds.), New Learning. Dordrecht, The Netherlands.: Kluwer.