Designing Impact Evaluations: What are the Appropriate Research Questions and Methods? Mywish K. Maredia Michigan State University Workshop for Managers of Impact Evaluation May 13, 2013 InterAction, 1400 16th St. NW, Suite 210 Washington, DC
Your Role in Explaining the ‘Miracle’ • As Impact Evaluation Managers, your role is to ensure that a plan is in place to bridge the knowledge gap between how project outputs result in impacts; and • Doing so based on evidence generated using credible methodology
Focus of this presentation • Discuss the development of appropriate impact evaluation questions • What types of impact evaluation designs are most appropriate in different contexts and given the evaluation questions? Purpose: • To share some preliminary thoughts; present a framework; and • To facilitate discussion and exchange of ideas / experience
Clarifying the term: Impact evaluation What it is: • It is concerned with establishing a causal link between realized impacts (the effect) and an intervention (the ‘cause’) which could be a program, activity, policy change, etc. The goal of the analysis is to ‘rule out’ other possibilities / explanations for the observed effects
What do we mean by evaluation design? • Every evaluation is essentially a research or a discovery project/activity • If your results are to be reliable, you have to give the evaluation a structure that will tell you what you want to know • That structure – the arrangement of discovery- is the evaluation’s design • The design depends on what kinds of questions your evaluation is meant to answer
Development of Impact Evaluation Questions Characteristics of ‘appropriate’ IE questions: • They should be narrow/specific • Focus on small number of questions (~5) • Focused on ‘summative’ evaluation of a project/intervention • They should reflect the input of program staffand sponsors
Examples of common impact evaluation (research) questions Overall impact (effectiveness) • Did it work? Did the intervention produce the intended impacts in the short, medium and long term? • For whom, in what ways and in what circumstances did the intervention work? • What unintended impacts (positive and negative) did the intervention produce? Source: Rogers (2012) Introduction to Impact Evaluation. Impact Evaluation Notes No.1
Examples of common impact evaluation (research) questions (cont’d) Nature of impacts and their distribution • Are impacts likely to be sustainable? • Did these impacts reach all intended beneficiaries? Influence of other factors on the impacts • How did the intervention work in conjunction with other interventions, programs or services to achieve outcomes? • What helped or hindered the intervention to achieve these impacts?
Examples of common impact evaluation (research) questions (cont’d) How it works • How did the intervention contribute to intended impacts? • What were the particular features of the intervention that made a difference? • To what extent are differences in impact explained by variations in implementation? Matching intended impacts to needs • To what extent did the impacts match the needs of the intended beneficiaries?
Common impact evaluation designs (focused on causal analysis) Methods for examining the factual. For e.g.: • Comparative case studies • Beneficiary/expert attribution Methods for creating counterfactual • Experimental designs or RCTs (based on the principle of random assignment) • Pipeline comparisons • Other methods/approaches (using statistical techniques to form credible comparison groups). For e.g.,: • Propensity score matching (PSM) • Instrumental variables (IV) • Regression discontinuity (RD) • Difference in difference (DD)
IE Methods • In theory, there are multiplicity of methods and approaches that can be used to assess impacts • Each have problems and limitations • There is no ‘one size fits all’ method/approach
Choosing an appropriate impact evaluation design Depends on… 1. The nature of the research questions
Choosing an appropriate impact evaluation design (cont’d) Depends on… 2. The nature of your program
Choosing an appropriate impact evaluation design (cont’d) Depends on… 3. what participants / stakeholders will consent to? 4. What are your resources and time constraints?
Practical Considerations in Designing Impact Evaluation • Establishing the program theory (the logic behind a program and the causal chain from inputs to outcome to impact) • Understanding the program setting • How participants are selected (to mitigate selection bias) • Decision tree on which method is applicable and should be explored
Practical Considerations in Designing Impact Evaluation (Cont’d) • Sample size • Power calculation – Does the setting allow for enough numbers of units of intervention and units of observation for a robust design? • Tradeoff between power and cost • Time frame • Is there enough time to observe the impact? • Flexibility • Strive for rigor, but be flexible…