Participatory Approaches in Impact Evaluation

1. Participatory Approaches in Impact Evaluation AsliGurkan Social Development Department World Bank Dubai – Impact Evaluation workshop May 31-June 4 Social Development Department The World Bank

2. Setting the Context • Heightened attention to governance issues at the World Bank since adoption on GAC strategy • Increasing emphasis on outcomes/impact to enhance development effectiveness • More attention to transparency, access to information, citizen-participation at all levels in World Bank operations (mandatory with Investment Lending reform) • Key focus of Social Development department-strengthening demand-side of governance (including PM&E methods)

3. What is demand for good governance (DFGG)? • The ability of citizens, civil society organizations and other non state actors to hold the state accountable and to make it responsive to their needs • In turn, DFGG enhances the capacity of the state to become transparent, accountable and participatory in order to respond to these demands • DFGG mechanisms can be initiated and supported by the state, citizens or both • but very often they are demand-driven and operate from the bottom-up. The people have a right to know, a right to question, a collective Constitutional right to receive an answer.” Aruna Roy, MKSS Rajasthan, India

4. Key Demand-side mechanisms to address governance challenges • Transparency/Access to Information • Consultation/Participation • Grievance Redress Mechanisms • Third party Monitoring/ Independent verification of outcomes • Participatory Impact Assessments

5. Possible methods/tools under PM&E: • Visual techniques • Activity monitoring chart • Participatory Rural Appraisal • Citizens report cards • Community scorecards • SARAR (participatory problem solving tool) • Participatory Impact Assessments

6. What’s Participatory Impact Assessment? “Involves the adaptation of participatory tools combined with more conventional statistical approaches specifically to measure the impact of humanitarian assistance and development projects on people’s lives.” Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners

7. Participatory versus Conventional IE Source: Adapted from DeepaNarayan, World Bank.

8. At the project level-3 key questions… • What changes have there been in the community since the start of the project? • Which of these changes are attributable to the project? • What difference have these changes made to people’s lives?

9. Designing and Implementing PIA: Steps Stage 1: Define Questions Stage 2: Define the geographical and time-limits of the project Step 3: Identify locally defined impact indicators Step 4: Decide on ranking/scoring methods on and testing sampling methods Stage 5: Choose Sampling Methods Stage 6: Assess project attribution Stage 7: Triangulation Stage 8: Feedback and verify results with community Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners

10. Step 2, Tool #1: Defining the project boundary: participatory mapping A map of Zipwa Site, Zimbabwe Community members drawing a map in the sand

11. Step 2- Tool # 2: Define the project period by timelines established by the communities • Creating a timeline-- • - Identify a Knowledgeable person (or persons) in a community • Ask them to describe the history of the community. • - In many rural communities, such descriptions usually refer to key events such as drought, periods of conflict or disease epidemics • The project start and end time should be related to these key events.

12. Step 3: Indicators-identified by communities themselves… Ex: Drought projects in Zimbabwe and Niger Tips for practitioners: Make sure to capture the views of different groups of people within the community. (Women will often have different priorities and expectations of project impact than men.)

13. Step 4: Methods for Scoring/Evaluation Tool #1: Scoring of Food sources using counters- Evaluating the impact of a community garden project participants identify all the food sources that contribute to the household food basket. Practitioner Tips- Where informants are literate you may choose to simply write the name of The indicator on a card.

14. exercise -done with project participants for the agricultural year before and after the project and again for the agricultural year. The exercise then repeated with community members who had not participated in the project Monthly household utilization of the harvested maize until depletion (using 25 counters) Step 4-Tool # 2: Impact calendars-post-harvest food balance

15. Challenges with community-level PM&E approaches (from Bolivia and Nepal) • Clash of incentives: pressure from donors to ‘prove’ impacts vs. adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations • Resistance from the project teams to changing their existing M&E practices • Project deadlines prevent project-staff from establishing a consistent PM&E practice • Availability of PME expertise within the field staff to facilitate the PM&E exercises, • Insufficient transfer of community-level PME skills to interest groups and grassroots organizations • Lack of sufficient training/capacity-building programs • Human resource problems and lack of capacity in analyzing qualitative data and reporting results.

16. Possible recommendations to improve PM&E activities • Develop ready-to-use templates, a detailed Community Researcher manual to improve the research data and reports. • Conduct regular follow up visits to each case study sites to review the work of the Community researchers and provide feedback, advice and support • Identify mentors, encourage the community researchers to phone their mentors on a regular basis to share their progress and any problems they had. • Encourage the community researchers to contact each other regularly to share their experiences and reports. Promote “peer-to-peer learning’

17. Key messages • Keep participatory evaluation methodologies simple and practical • Develop your methods, standardize and repeat. • the more repetitions, or the larger the sample size, the more statistically reliable the results will be. • Remember to field test your methods with community members before the assessment–most methods look easy on paper but require fine tuning once you start to use them in the field.

18. Useful links/Resources Participatory Impact Monitoring Booklet I- V GTZ Feinstein International Center-Tufts University Participatory Impact Assessment: A Guide for Practitioners World Bank Participation and Civic Engagement Website: World Bank Social Accountability Sourcebook

19. Thank you