Data Collection Methods

Data collection is a specialist subject and we are not proposing to deal with the complexities of survey design or the individual methodologies. However, virtually all evaluations will involve collection of some primary data and you may have ideas about the sort of methods you think would be useful and appropriate to your organisation or project.  The methods most likely to be used will include data from questionnaires, interviews, focus groups and discussion groups, review meetings, conferences and seminars, structured evaluation exercises and activities and direct observation of project activity.
The secondary data sources may include a subject area literature review, institutional, local and national and international policy and strategy documents, other project evaluation outputs, working papers and minutes of meetings and project reports, project application documents, demographic survey reports and so on.
If the evaluation is using performance indicators then these will determine what data needs to be collected. That is, if there are performance indicators based on the number of trainees, number of web site hits, number of new businesses or similar, these things will obviously have to be counted and ways of counting them will have to be designed. However, not all evaluation, especially ‘developmental’ evaluation, will use performance indicators.
There are basically two different, broad approaches to data collection for evaluation or at least, two ends of a continuum.

Process driven data (PdD)

In this case, the data to be collected is categorised in advance and processes are set up to collect particular sorts of information. This will always be the case where the evaluation is using performance indicators. In the interests of efficiency, the collection process is largely designed around what data is to be excluded and to eliminate unusable responses.
This has the obvious advantage of being able to focus the evaluation very precisely but the disadvantage is that possibly important issues will be excluded accidentally by the perceptions and assumptions of those who determine the data categories.

Data driven process (DdP)

Within the limitations of the overall dimensions of performance to be evaluated, this is a more ‘broad brush stroke’ approach that does not categorise the data rigidly in advance. Data is collected in ways that are often deliberately constructed to be as open-ended as possible and then examined for patterns or recurring themes.

Example

Consider a simple training course evaluation undertaken by a teaching team. It may be that they are considering changing the audio-visual technology or house style of the handouts or similar. The course may provide an opportunity to collect some detailed data though the use of highly structured closed question questionnaires.
E.g. “Was the project screen clearly visible at all times? Please mark on the seating plan where you were sitting.”
Conversely, there may be a ‘sentence completion’ type exercise or group discussion on “What did you enjoy most on this course?” or “What was the best thing on this course?”   If the issue was to check on the usefulness of the audio visual aids this is hardly likely to generate useful information – unless they were so good or bad that participants felt driven to comment!
What is more likely, is that this sort of question will generate a whole range of (often-unexpected) responses.  For example, some may say the food was awful, some will say it was excellent; some will say that “meeting other people” was good, (or that there was insufficient time allowed for this) and so on.
The conclusions to be drawn, or lessons learned, therefore, if the responses are clustered, maybe that e.g. ‘catering facilities’ or ‘social interaction’ are important issues for participants.
Methods can be combined, and are obviously not exclusive, but what is important is to recognise that “facts” and ‘issues” will need different collection processes and different tools.