By Jenny Hughes

The Evolution of evaluation theory

Evaluation as a formal activity that we would recognise, has existed for a surprisingly long time. One of the earliest recorded was the evaluation of the effectiveness of lime-juice in preventing scurvy in sailors – commissioned by the British navy in the 18th century! The French make even earlier claim and say that the Norman armies conducted an evaluation of the relative effectiveness of the crossbow and the longbow. Unfortunately, on the basis of the evaluation findings, the management decision was to go for the crossbow and the rest, as they say, is history!

However, evaluation has only become a recognised area of academic study since about the 1960’s. It is probably true to say that evaluation started as a field of practice and the theory was derived from it. As it evolved, so ideological disputes developed alongside disagreements on definitions, terminology, ethics and so on. FitzPatrick, Sanders and Worthern in 2004 identified nearly 60 different models in the 30 years between 1960 and 1990 alone. This proliferation of models was bewildering for the practitioner, especially as many of these models and the tools they generated had no obvious theoretical perspective.

Why is this a problem? Why should practitioners need a theoretical framework? Simply, a ‘good’ theory will set out the assumptions that it is making and on which its logic is predicated. Different theories make different assumption and generate models that will be based on different pre-conceptions and definitions of evaluation, which in turn lead to very different practices.
Deriving a taxonomy of evaluation approaches.

Many researchers have tried to make sense of this huge diversity of models and theories and to find some way of classifying them. However, even they could not agree so now we have the problem of trying to classify the classification systems!
All this is by way of saying that what follows is only one framework for distinguishing between different theories of evaluation and you may well come across others. This framework, which we find comprehensive and useable, was devised by FitzPatrick, Sanders and Worthen (2004) who we have quoted at length.

Philosophical /ideological differences

Approaches to evaluation may differ fundamentally because their underpinning philosophy or ideological base is different. That is, different evaluation theories will be based on different assumptions about the way the world works and so the models and practices based on those theories will be different as well. By and large, we can locate them on a continuum from objectivist to subjectivist.
Objectivism is equivalent to the empirical tradition in scientific research (positivism) and focuses on data collection and analysis techniques that produce results that are reproducible and verifiable by other evaluators and to generate conclusions that are evidence based and which can be ‘scientifically’ justified. So the evaluation is ‘external’ to the evaluator who is simply someone technically competent and proficient in the application of procedures.

Subjectivism is based on:
“...an appeal to experience rather than to scientific method. Knowledge [of the evaluator] is conceived as being largely tacit rather than scientific.”
(House 1980 in FitzPatrick, Sanders and Worthen 2004)

The validity of a subjectivist evaluation depends on the relevance of the evaluators’ background, their experience and expertise, the keenness of their perceptions and their insightfulness in generating interpretations and conclusions. Thus, the evaluation procedures are ‘internal’ to each evaluator and are not explicitly understood or reproducible by others.

Until 20 years ago, objectivism in evaluation was a goal to be aspired to. However, the same criticisms levelled at the usefulness of positivism in the social sciences in general were also applied to objectivism in evaluation.

Campbell (1984) summed it up:
“twenty years ago positivism dominated the philosophy of science...Today the tide has completely turned among the theorists of science in philosophy, sociology, and elsewhere. Logical positivism is almost universally rejected.”

This point of view has been upheld by many writers on evaluation and even if it is not universally subscribed to, probably represents a general trend. The major argument is that unlike traditional scientific research, evaluation has to deal with complex phenomena in real world settings, take into account a multiplicity of stakeholders, unstable and unpredictable systems and requires a high level of human interactivity.

The other criticism is that objectivism depends for its validity on its ‘scientific’ methodology and is only credible if you happen to value that methodology. We would argue that objectivism conceals hidden values and biases of which many evaluators are unaware – even the choice of data collection techniques and instruments is not value-neutral but this is not recognised or else ignored by many so-called objective evaluations.

Despite the reaction of the theorists, however, the message does not seem to have filtered through to the client base and the majority of evaluation consumers, particularly in education (and the public sector in general), are still wedded to the idea of objective evaluation and ‘finding the facts’.

The major criticism is that subjectivist evaluation often leads to contradictory conclusions that cannot be reconciled because the processes which led to the conclusions is largely inside the evaluators head and so cannot be replicated.

Differences based on defining value or worth

We can also distinguish between different theoretical approaches depending on how they define value and make judgements, rather than on their philosophical differences. This time the continuum extends from ‘utilitarian’ to ‘intuitionist-pluralist’.

‘Utilitarianism’ is a philosophy based on maximising happiness in society. Utilitarian approaches to evaluation are based on the premise that the best programmes are those that achieve the greatest good for the greatest number. The evaluator will try and assess overall impact in terms of total group gains by using average outcome scores against the criteria selected to determine worth. Again, governments and the public sector tend to be adherents of this type of evaluation as it lends itself to large-scale comparisons of programmes and mass aggregation of data. Managers and public programme administrators tend to be the main audiences.

According to FitzPatrick et al, the intuitionist-pluralist approach is at the other end of the spectrum and is based on the premise that value depends on the impact of a programme on each individual and the ‘greatest good’ is that which maximises the benefits for all stakeholders. “This evaluation focus will be on the distribution of gains (for example by cultural or sub-cultural demographic groups such as ethnicity or gender or age) or distribution of benefit across stakeholders (e.g. learners, administrators, delivery agencies, funding bodies, the public). There can be no common index of “good” but rather a plurality of criteria and judges. The evaluator is no longer an impartial ‘averager’ but a portrayer of different values and needs. The merit or worth of any programme depends on the values and perspectives of whoever is judging it and all stakeholders are legitimate judges.”

Methodological differences

Although there is a strong correlation between an evaluator’s ideological approach and the methodology and techniques they will use (because of necessity one drives the other), there are other major divides based on methodological differences that are not necessarily rooted in a particular philosophical approach. For example, many evaluators (both theoreticians and practitioners) and also many clients tend to view qualitative and quantitative approaches as different paradigms.  We do not subscribe to this view, believing that this is not a fundamental divide but simply a way of describing evaluation approaches by types of data that are used. Nevertheless, we recognise this as an important distinction for others and one that impacts on the overall evaluation methodology and the tools used.

Differences according to discipline or field of application

Evaluation is a relatively young field and still draws heavily on methodologies adapted from anthropology, sociology, psychology, philosophy, economics and mathematics. One of the consequences is that evaluation approaches can be grouped around their parent discipline so we tend to find ‘mathematical approaches’ or ‘sociological approaches’. More recently the search for new models has widened its net and evaluation theorists such as Smith (1981) are trawling newer disciplines such as investigative journalism, photography, storytellling, philosophical analysis, forensic pathology and literary criticism for new ideas.

Evaluation theory has also developed in a social context and practitioners work in different cultures, different sectors, with different target groups and different audiences. Consequently, different approaches and models have tended to emerge based on these factors. For example, ‘education programme’ evaluation has developed along a different trajectory than, for example, the health services. Whilst many writers would argue that this is not a true theoretical divide, ‘theory-in-practice’ is a powerful determinant of evaluation approach and also stakeholders perceptions and expectations of the evaluation process.

Differences in practice.

The above distinctions are all based (loosely) on theoretical divisions. However, FitzPatrick et al also point out that differences in approach can be practice-driven.

Firstly, evaluators disagree about whether evaluators should simply provide information so that decision makers can make the value judgements. Others, would say that the evaluator’s report to decision makers is incomplete if it does not contain a value judgement.
Secondly, evaluators differ in their perception of their own role and their place in the evaluation process. Who has authority and responsibility for the evaluation and to whom should the evaluator be accountable and answerable? If one evaluator sees his role as a ‘critical friend’ and another as ‘inspector’ or ‘judge’, then this will obviously influence the way they conduct an evaluation and also the conclusions they draw.

Thirdly, evaluators will be limited by their prior experience both in evaluation and also by their own discipline or professional background. Evaluation skills and knowledge are cumulative. Previous exposure to frequently recurring problems will affect the way an evaluator works. On the one hand it will probably mean the evaluator is quicker to detect problems, to identify issues of concern and make more insightful judgements. On the other hand, it will also mean that the evaluator’s perceptions in a new situation are unlikely to be ‘neutral’.

Fourthly, evaluators have different views about what skills and expertise they should possess. Evaluators are frequently chosen on the basis of their expertise or practitioner base in the field being evaluated rather than on the basis of their skills and experience as an evaluator. This is gradually changing but as evaluation is becoming increasingly professionalised and recognised as a specialist area in its own right, so professional evaluators are becoming specialised within the area. Some evaluators would argue that specialist knowledge of the field being evaluated is a pre-requisite for the credibility of the whole process of evaluation. Others claim that not only is this unnecessary but can, on occasions, be unhelpful.

A classification system

The above analysis is interesting and helps understand the major theoretical divides in evaluation. However, it does not get us far in terms of systematically examining the variation between particular evaluation approaches because although those approaches could be positioned on each of the above ’dimensions’, their location would vary from one dimension to another. The next section tries to provide some answers.

Many evaluation theorists have attempted this but we are going to stick with the solution put forward by Fitzpatrick, Sanders and Worthen (1983). We are proposing to use their work – with some modifications - partly in the interests of consistency (having referenced them heavily so far) and partly because they set out very clearly the thinking and rationale underpinning their classification system.

For the purist it is a less than satisfactory taxonomy as the approaches do not necessarily differ from one another along the same dimension. However, they are pragmatic as they conveniently represent the major clusters of models and approaches in use today.

A Classification Schemata for Evaluation Approaches

FitzPatrick et al identify 5 major clusters of evaluation approaches:

• Objectives oriented approaches
• Management oriented approaches
• Consumer oriented approaches
• Expertise oriented approaches
• Participant oriented approaches

However, to this we propose to add Van der Knapp’s ‘learning oriented approach’.

These 6 categories fall more or less along a continuum from utilitarian to intuitionist-pluralist so there is some logical basis in addition to its convenience and accessibility (see figure 1).

Objectives orientated evaluation approaches

Objectives-orientated evaluation is based on the idea that the purposes, goals or targets of a project are determined at the start and the evaluation process should establish whether these have actually been achieved – and, if not, why not. It is very similar to another approach known as ‘a systems approach’ to evaluation and both are very popular with public sector agencies who are concerned with justifying expenditure and performance measurement. It is sometimes called ‘goal-driven’ evaluation, in contrast with other approaches, which are called ‘goal-free’.

There are many examples of objectives orientated models; the earliest is probably Tyler’s and more recently, Provus’s Discrepancy Model.

The disadvantages are that this sort of approach can miss important outcomes if they were not included in the original objectives nor does it challenge the value of the objectives themselves.

Management orientated evaluation approaches

The management-orientated approach to evaluation is meant to serve decision makers. Its rationale is that evaluation information is an essential part of good decision making and that the evaluator can be most effective by focussing the evaluation products on the needs of managers, policy-makers, administrators and practitioners.

Developers of this approach have traditionally relied on a systems approach to evaluation in which decisions are made about inputs, processes and outputs based on logic models and cybernetic theory. However, more recent developments have highlighted different levels of decision and decision makers and have focussed on who will use the evaluation results, how they will use them and what aspect(s) of the system they are making decisions about.

Not surprisingly, it is the model preferred by many managers and management committees but the downside is that the needs of other stakeholders are ignored.

Stufflebeam’s CIPP model is one of the most popular in management-orientated evaluation.

Consumer orientated approaches

Consumer orientated approaches to evaluation adopt the perspective of the end user of whatever service or product is being provided.  For this reason they tend to be summative, rather than formative and are concerned primarily with product evaluation. Consumer-orientated evaluation relies heavily on criteria referenced evaluation techniques such as benchmarking or kite marking and is understandably popular with standards agencies and ‘watchdog’ organisations. Michael Scrivens ‘Key Evaluation Checklist’ is probably the best-known example.

The major disadvantage of a consumer-orientated approach is that it is a ‘backward-mapping’ approach and does not help make predictions about future impacts. It also tends to play down the nature of human interaction with the products being evaluated.

Expertise orientated approaches

Expertise orientated evaluation is based on the notion of “connoisseurship” and criticism and relies on the subjective professional judgement and expert knowledge of the evaluator. This is the oldest form of evaluation and is still very popular despite its limitations.
Expertise-orientated evaluation may be formal or informal, based on individual expertise or, more usually, on the collective expertise of a panel. The opinions of multiple experts is popularly believed to minimise bias, though this does not always follow! It relies far less on external tools and instruments than other forms of evaluation and more on the experience and wisdom of the evaluator.

Many public systems are based on expertise orientated evaluation – for example the jury system, school inspection system, licensing agencies, review boards, the refereeing system for academic journals, national commissions and enquiries and so on.

Many organisations expect this type of evaluation if they employ an external evaluator and the notion of evaluation by ‘peer review’ is still the dominant model in most professional associations.

The disadvantages are obviously the high reliance on the assumed expertise of the evaluator and a lack of explicit and published standards. Also, the credibility of results is affected by the status of the evaluator but equally the credibility of the evaluator is often affected by the results.

Learning-orientated evaluation approaches

This is a relatively new group of approaches and not one that was included in FitzPatrick et al’s classification. Nevertheless we have included it because it is an approach that we personally use more than any other.

The operating principle is that the purpose of evaluation is to contribute to some form of collective or organisational learning. Different models within this overall approach are based on different theories and types of learning including ‘corrective’ or behavioural learning, cognitive learning and social learning. The outputs and processes of the evaluation form the inputs of the learning.

The pioneer of work in this field was Peter Van der Knaap. More recently we have extended the approach to include evaluation as a contributor to knowledge creation in an organisation.

Learning-orientated evaluation approaches are still not widespread but are beginning to gather momentum in the social agency sector, in education establishments and in voluntary organisations.

The main limitations of this approach is that it does not lend itself to “mass surveys” as it relies heavily on personal interaction between the evaluator and the project team and the evaluator’s understanding of the learning needs of the organisation. Also, within this overall approach there are very disparate models, some requiring a high level of commitment to the process, which may be lacking.

Participant-orientated evaluation approaches

An increasingly popular approach that differs fundamentally from all the others as it takes the needs of project participants as its starting point. This is not to say that the other approaches ignore participant needs but that for the most part benefits for participants represent the end point of the evaluation and not the beginning.

Participants are not simply the direct beneficiary target group of a project but will also include other stakeholders and potential beneficiaries. Thus, an educational project for women returners would include the learners themselves, the project staff, the management team and the funders but may also include the wider community, the learners families, the schools attended by the learners’ children, childcare agencies or whatever.

Participant-orientated evaluation does not usually follow a formal plan drawn up in advance; rather it looks for patterns in the data as the evaluation progresses. Data is gathered in a variety of ways, using a range of techniques and culled from many different sources. Understandings grow from observation and bottom up investigation rather than rational deductive processes. The evaluator’s role is to represent multiple realities and values rather than singular perspectives.

Participant-orientated evaluation includes many sub-groups that share all or some of the above characteristics including Responsive Evaluation, Naturalistic Evaluation, Utilisation Focussed evaluation and Empowerment Evaluation. Of all the models, probably the best known and one of the most useful is Stakes Countenance Framework.

Criticisms of this approach are many; bureaucrats tend to hate it because of its lack of ‘objectivity’ and because the outputs of the evaluation are unpredictable. It is difficult to cost and control. Without a very experienced evaluator to facilitate the process, it can degenerate from an ‘organic’ approach to one which is chaotic and lacking in focus. Also, there may be concentration on process at the expense of outputs.

References

Administration on Children, Youth and FamiliesThe Program Manager's Guide to Evaluation, http://www.acf.dhhs.gov/programs/core/pubs_reports/foreword_pmguide.html, accessed 4 November, 2003

Allum, N.C Bauer, M.W., Gasgell, G. and. (2000) Quality, Quantity and Knowledge interests, Avoiding confusion in Bauer and Gasgell, Qualitative research, Lonmdon, Sage

Boone, E. J., Developing Programs in Adult Education, Englewood Cliffs, New Jersey: Prentice-Hall Inc., 1985.

Boyle, P. G., Planning Better Programs, New York: MacGraw-Hill Book Company, 1981.

C.L.Taylor and Carl E. Beeman2 (1992) Evaluation for Accountability: An Overview, University of Florida

Describing Innovation as a prerequisite for innovation, Hughes and Attwell 2000, www.evaluate-europe.net

Donald Clarke (1997) Instructional system development: evaluation phase, http://www.nwlink.com/~donclark/hrd/sat6.html, accessed November 4, 2003

Israel, Glenn D. and C. L. Taylor. 1990. "Can Response-Order Bias Evaluations?" Evaluation and Program Planning. 13(4):365-371
Mendenhall, W., and R.L. Scheaffer. 1973. Mathematical statistics with applications. Duxbury Press, North Scituate, MA.

Patton, M. Q. (1986) Utilization-focused evaluation. Newbury Park, CA: Sage.

Project Star, User's Guide to Evaluation for National Service Programs, http://www.projectstar.org/star/Generic/usersguide.htm, accessed November 4, 2003

Rossi, P. H., & Freeman, H. E. (1989). Evaluation: A systematic approach. Newbury Park, CA: Sage Publications.

Ryans, D. G. (1960). Characteristics of teachers. Washington, DC: American Council on Education

Scrivens M. (1991), The  evaluation Thesaurus, Sage

Weiss The Interface Between Evaluation and Public Policy," in Evaluation: The International Journal of Theory, Research, and Practice (1999)