Participatory approaches to evaluation

1. Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk

2. Session outline • What is Participatory evaluation? • Philosophical underpinnings • Barriers to participatory approaches • Participatory Action Research (PAR) • Most Significant Change technique (MSC)

3. Participatory evaluation-1 “Participatory evaluation is a partnership approach to evaluation in which stakeholders actively engage in developing the evaluation and all phases of its implementation” -Zukoski and Luluquisen (2002)

4. Participatory evaluation-2 “Participatory monitoring and evaluation is not just a matter of using participatory techniques within a conventional monitoring and evaluation setting. It is about radically rethinking who initiates and undertakes the process, and who learns or benefits from the findings.” -Institute of Development Studies (1998)

5. Participatory evaluation-3 “Participatory evaluation means more than just a way of seeing how much has been achieved or produced at what cost and with what effect…Participatory evaluation is not just to do with the development of things. It is to do particularly with the development of people.” -Feuerstein (1986)

6. Historical and Philosophical basis “The basic ideology of PAR is that self-conscious people, those who are currently poor and oppressed, will progressively transform their environment by their own praxis. In this process others may play a catalytic and supportive role but will not dominate" -Fals-Borda, 1991

7. Freirean approach to community development • Paulo Freire (Pedagogy of the Oppressed) • An educational philosophy that actively involves the poor in critically analysing their social situation (conscientisation), thus creating the potential for them to transform their environment. • Praxis: reflection-action dynamic

8. Group reflection • How participatory should participatory evaluation be? • What are the advantages of using participatory approaches to evaluation? • Can a participatory evaluation replace traditional evaluation methods? • Are participatory approaches to evaluation valid? • What are the barriers to participatory approaches?

9. Participatory Action Research (PAR) “Put simply, action research is ‘learning by doing’ - a group of people identify a problem, do something to resolve it, see how successful their efforts were, and if not satisfied, try again.” -Rory O’Brien (1998)

10. The action research process • Identify problem • Collect data for detailed diagnosis • Collective postulation of possible solutions • Agree and implement an action plan • Collect data on results of intervention • Analyse data • Interpret findings in relation to success of action • Re-assess problem • Begin another PAR cycle

11. Principles of action research (Winter 1989) • Reflexive critique • Dialectical critique • Collaborative resource • Risk • Plural structure • Theory, practice, transformation: theory informs practice and practice refines theory

12. Application of PAR in health • Health Needs Assessment • Planning health care services • Service evaluation • As a process of empowerment • Can employ both qualitative and quantitative methods • Qualitative tools include: journals, interviews, focus groups, filming, use of pictures or drawings • Quantitative tools: surveys, monitoring data

13. RAP, RRA, PRA, PLA • Rapid Assessment Procedure (RAP) • Rapid Rural Appraisal (RRA) • Participatory Rural Appraisal (PRA) • Participatory Learning and Action (PLA)

14. RAP techniques for data collection • Proportional piling: visual form of estimating proportions. • Community maps: Drawn in groups; Useful to learn about an area and how different groups use it • Transect walk: observation technique • Diagramming: Venn (chapati) diagrams to represent power structures (power represented by size; relationships by overlap; exclusion by distance)

15. Most Significant Change Aka ‘Monitoring-without-indicators’, ‘the story approach’, ‘evolutionary approach to organisational learning’…

16. The kernel of the MSC process • A question: “Looking back over the last month, what do you think was the most significant change in [particular domain of change]?” • A second question: “From among all the significant changes, what do you think was the most significant change of all?”

17. Steps • Start and raise interest • Define domains of change (e.g. quality of life, health, participation/empowerment) • Define reporting period • Collect significant change (SC) stories • Select the most significant of the stories (hierarchical level) • Feed back results of the selection process • Verification of stories • Quantification • Secondary analysis and meta-monitoring • Revising the system

18. When to use MSC • Complex programmes with diverse and unexpected outcomes • Focussed on social change • Participatory in ethos • Highly customised services • Focus on learning rather than just accountability • Enablers: organisational culture of ‘no-blame’; willingness to try something different; commitment by senior managers

19. Defining domains of change • Domains are broad and often fuzzy categories of possible SC stories • A domain of change is not an indicator (deliberately fuzzy as opposed to SMART) • Include an ‘open window’ domain • Could include a domain for negative changes ‘lessons learned’ or ‘changes that reflect an area to improve’ • Could be based on programme objectives • Between 3-5 domains are best

20. Collecting the SC stories • Information to be documented includes: • Who collected the story and where the events occurred • Description of the story itself- what happened (sequence of events) • Significance (to the storyteller) of the events described in the story • Stories could be 1-2 pages long

21. Selecting the most significant of stories • Hierarchy of selection processes • Criteria for selecting stories: Majority vote, iterative voting, scoring • Documenting the results of the selection process: the reasons for selecting an SC story as the most significant should be documented and attached to the story

22. Secondary analysis and meta-monitoring For all collected stories: • Thematic coding • Look for positive and negative changes • Analyse changes mentioned in MSC stories against a logic model • Analyse genre • Analyse differences between selected stories and those not selected

23. What can MSC offer as an evaluation technique? • Uses an inductive approach to help identify unexpected outcomes • Helps monitor messy impacts, intangible and indirect consequences • The hierarchical selection process of sorting through diverse SC stories becomes an opportunity for making implicit values explicit, communicating shared values and agreeing a direction for the project

24. Validity and voice in MSC Thick description: richness of detail and context Systematic process of selection (compare to case studies) Transparency (systematically records interpretation) Verification Participation in keeping with the broader programme philosophy of empowerment Purposeful sampling

25. Further reading 1. Davies, R. and Dart, J. (2005).The ‘Most Significant Change’ (MSC) Technique: A guide to its use (http://www.mande.co.uk/docs/MSCGuide.pdf) 2. Action research resources (http://www.scu.edu.au/schools/gcm/ar/arhome.html)