html5-img
1 / 14

Introduction to Using Qualitative Methods in Impact Evaluation

Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1, 2010. Introduction to Using Qualitative Methods in Impact Evaluation. Multiple Reasons for Using Participatory Methods .

evette
Télécharger la présentation

Introduction to Using Qualitative Methods in Impact Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1, 2010 Introduction to Using Qualitative Methods in Impact Evaluation

  2. Multiple Reasons for Using Participatory Methods • This morning’s session emphasized social justice and accountability as reasons to utilize participatory methods in impact evaluation • There are are also methodological justifications for integrating participatory assessment and other qualitative measures with randomized impact evaluation

  3. What Are Qualitative Methods? • Qualitative methods are a diverse set of research tools that are primarily concerned with the meaning of social actions, statements, and documents • Both alone and in combination with randomized evaluation and sample surveys, these methods provide leverage with respect to: • Describing and categorizing social contexts • Interpreting the social behavior, actions, and outcomes • Tracing the specific processes and mechanisms whereby programs lead to effects

  4. The Diversity of Qualitative Methods • Participatory and Action-Based Research • Interviewing and Discussion • Content Analysis of Documents and Records

  5. The Diversity of Qualitative Methods • Participatory and Action-Based Research • Systematic Social Observation • Drama and Role Playing • Maps, Models, and Diagrams • Ranking and Scoring • Interviewing and Discussion • Content Analysis of Documents and Records

  6. The Diversity of Qualitative Methods • Participatory and Action-Based Research • Interviewing and Discussion • In-Depth, Semi-Structured Interviews • Focus Group Discussions • Oral Histories of Individuals, Communities, or Projects • Content Analysis of Documents and Records

  7. The Diversity of Qualitative Methods • Participatory and Action-Based Research • Interviewing and Discussion • Content Analysis of Documents and Records • Diaries or Personal Time/Financial Records • Archives and Government Records • Newspapers and Civil Society Reports

  8. Analyzing Qualitative Data: The Role of Narrative • Despite the differences between these methods of collecting qualitative data, the common principle is using close observation to understand the meaning of what people say and do • As a result, the writing of field notes and systematic the aggregation of notes into narrative case studies are a central skill in analyzing qualitative data

  9. Combining Qualitative and Quantitative Methods • Qualitative methods can be linked to randomized evaluation • Before Implementation: • Use interviews, participation, andobservation to design contextually relevant surveys and behavioral activities • Use observation or participation to design interventions that can test the interaction of different program components • After Implementation: • Aggregating and coding the content of interviews, behavioral observation, and newspapers to cross-check survey data • Analyzing process to understand the lack of statistically significant differences • Implementation failure vs. design failure?

  10. Homework: Qualitative Methods and Your Projects When you meet with your clinic groups this afternoon, consider this question: What types of qualitative methods would add value to the evaluation of your project? Focus on three things: • Identify each question you want to answer • Choose a set of qualitative methods to integrate into your evaluation design for each questionand how/when you would useeach method • Why these methods add value to your research design

  11. Qualitative Methods:Methodological Limitations • Small, Nonrepresentative samples • Researcher/Facilitator Bias • Possibility of Contamination of Control Groups when combining with Randomized Evaluation • Lack of standards for comparing qualitative and quantitative findings

  12. Qualitative Methods:Practical Limitations • Time and Cost of Qualitative Research • Understanding and Documenting Context Requires Time to Develop Mutual Trust Between Researcher and Community • Increased Staffing and Training due to more diverse skill sets • Coordination and Management Problems Associated With Larger Teams

  13. Conclusions: Mixing Qualitative and Quantitative Methods • Qualitative Methods are a diverse set of tools and skills that require training, practice, and careful research design • Qualitative methods can help to better understand why a given program did or did not have an effect • Helping to develop better treatments and outcome measures for randomized evaluations • Using localknowledgeand cultural context to interpret the meaning of social behavior and process of intervention • Ideal mix of qualitative and quantiativemethods depends on the questions you want to answer and the local context

  14. REFERENCES AND SUGGESTIONS FOR FURTHER READING Bamberger, Michael, VijayendraRao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications Barron, Patrick, Rachael Diprose and Michael Woolcock (2010) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press (in press) Chambers, Robert. 1997. Whose Reality Counts: Putting the First Last, ITDG Publishing. Cronk, Lee. 2007. “The influence of cultural framing on play in the trust game: a Maasai example” Evolution and Human Behavior. Kanbur, Ravi, ed . 2001. Q-Squared? Qualitative and Quantitative Methods of Poverty Appraisal Malan, N. 2000. “On the Relationship between participatory research and participatory development.” Africanus 2: 64-75. Ravallion, Martin. 2001. “How Can Qualitative Methods Help in Measuring Poverty,”Qual-Quant: Qualitative and Quantitative Poverty Appraisal- Complementarities, Tensions and the Way Forward, Permanent Black Publishers, 2002. Rodrik, Dani. 2008. “The New Development Economics: We Shall Experiment, but How Shall we Learn?” Paper Prepared for the Brookings Development Conference, May 29-30 2008. Sheely, Ryan. 2010. “Community Governance, Collective Action, and the Maintenance of Local Public Goods: Qualitative and Experimental Evidence from Rural Kenya.” Better Governance for Better Health Conference, Stanford University. April 26-27, 2010. Van derRiet, Mary. 2008. “Participatory Research ant he Philosophy of Social Science.” Qualitative Inquiry 14 (4): 546-565. White, Howard. 2002. “Combining Quantitative and Qualitative Approaches in Poverty Analysis,” World Development. Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14

More Related