Trainer’s outline

How do we make judgement based on data?

Approach

• Go through the approach of data to information to knowledge
• Reference back to original objectives and aims
• Differentiate measurement from judgement
• Outline pitfalls to be expected Philanthropoid fallacy, Hawthorne and Pygmalion Effects
• Uses of judgements

Making Judgements

‘Making Judgements’ follows a series of logical steps.
1. Converting the raw data into collected data.
This could include statistically ‘summarising’ the numerical data, grouping or clustering observations and comments from interviews etc. by similarity of content, source, positive or negative responses and other factors. As far as possible it should also include a summary of ‘outputs’, deliverables, performance against targets and other monitoring data.
2. Analysing the data
In terms of its reliability, validity, conditions under which it was collected, context and so on. There may well be factors which need to be taken into account at this stage for example, interviews undertaken at a particularly “low” or “high” point, background circumstances which could have affected the data (changes in staff).
3. Interpreting the data
By looking for patterns, themes, and explanations. It is important at this stage that precipitate conclusions are not drawn and the range of possible interpretations and explanations are considered. It also involves identifying probable cause and effect relationships, general trends and directions.  It may also include setting up some hypotheses, which can be tested, and it is at this point that gaps in the data are most often identified – that is, more evidence is needed in certain areas.
4. Drawing conclusions
How this is done will depend entirely on whether the purpose of the evaluation is developmental or to do with project justifications. What follows is largely concerned with evaluation for improvement although the section on Performance Standards and benchmarking has been included which will be of more use to project accountability-type evaluation.
The conclusions can be expressed in terms of all or any of:

• Lessons learned about the activities (content) and also the processes (form) of the project.
• Recommendations for future action for policy makers, researchers, promoters, practitioners etc.
• Prescriptions for future action for policy makers, researchers, promoters, practitioners etc.
• Suggestions for improvement.
• Identification of critical success factors and also problem areas
• Plans and also predictions for long term impact analysis.
• New insights and alternative meanings
• Contribution the project has made to the overall ‘market’ gap it set out to fill or to the original problem it tried to solve.
• Areas of new work
• Checking against the original reference base

It is also necessary to review performance against the original reference base.  That is:

• for CRITERIA REFERENCING – has the project achieved the criteria (external criteria, stated objectives, specified deliverables etc.);
• for NORM REFERENCING – How well has the project performed compared with others?
• For IPSITIVE REFERENCING – what progress has the project made within its own environment – how far has it come?

Draw out lessons on the following topics from the handouts
Hand out 1    MEASUREMENT VS. JUDGEMENT (Scrivens)
Measurement = the allocation of numbers to entities in a systematic and intendedly objective (i.e., independently testable by means of observation of a simple procedure) way.
Judgement = the assessment by some cognitive process* of the extent to which an entity has a certain property [* as opposed to mere perception. In order to call something judgement, the brain has to work to get to the conclusion, whether consciously (as with a calculation) or unconsciously (as when you become so skilled that it becomes 'automatic', e.g. judges of diving)]
Hand out 2    THE PHILANTHROPOID FALLACY (Scrivens)
One of the biggest problems evaluators face, especially in philanthropic organisations is that, by spending a lot of time with a program and its staff, the staff can become convinced that they are beyond the need for evaluation.
Hand out 4    ANALYSIS ISSUES
Review the Hawthorne and Pygmalion Effects
Mistrust authority! Don't accept conventional wisdom; question everything. But don't forget that parables are great teaching devices, and their effect and value is not dependent on their truth, at least not entirely.
Hand out 5    Uses of Program Outcome Findings
Internal
External
Hand out 6 HOW DO YOU PREVENT EVALUATION ANXIETY? (Scrivens)
This can be used as a good practice checklist