The Importance of Negative Evidence

1. The Importance of Negative Evidence Rob D. van den Berg May, 2013

2. Setting the Stage • Evaluations should enable “learning from mistakes” • Negative evidence (“what does not work”) could help us • Evidence movement focuses on positive evidence • If it does not work: no clue why, just stop funding • If it does work: no certainty on why, just increase funding • The nature of positive and negative evidence • A framework for integrating negative evidence: a Theory of No Change (Christine Woerlen)

3. Data Gathering by a Turkey • Every day the Turkey gathers data on food, water and security: • Safe and secure environment on the farm with a fence to keep wolves and foxes out • Food and water delivered by the farmer every day • Counterfactual: Turkey’s distant cousin lives in the wild and faces many uncertainties… • Often on the run from predators • Organized hunts • Food and water availability have wild fluctuations • High probability that life is good for a turkey on a farm

4. Turkey graphs Predator threats Water availability Food availability

5. Farm turkey Cut-off point: head of the Turkey Wild turkey Predator threats Water availability Food availability

6. Black swan event (NassimTaleb/Popper) • Many data points on food and water availability and predator threats • Positive proof that farm turkeys are better off than wild turkeys • Farmer cuts off the head of the farm turkey • One event proofs that the “naïve” theory is not correct • Large n provides statistically significant proof for a theory • One n (a black swan event) is more powerful than the combined might of many n • However large the n is, it will never deliver 100 percent proof • One n may proof the theory wrong with 100 percent certainty

7. Causality in Science • Causality refers to the relationship between two events: the cause and the effect, where the second event is caused by the first • Scientific theories predict and explain effects • Early 20th century: logical positivism • Logic guides deductions from general theories to set up tests • Empirical data can provide positive proof of theory • Popper: logical positivism cannot escape the induction problem of Hume • However many data you gather, it will never constitute positive proof that the theory is right • The proof that is scientifically and logically sound is negative proof • Challenge is to falsify a theory

8. Positivism in Development • Logical positivism is no longer in vogue in the natural sciences • Testing of medicine is based on logical positivism and has been adopted as the “gold standard” by the evidence movement • Naïve positivism has been replaced with nuanced positivism that poses a null hypothesis that should be disproved; however, this still delivers “positive” proof the treatment works • Health, Education and Economics are heavily influenced; development has followed • Large n, divided in two groups (with/without intervention) is needed for evidence

9. From zero to small n • Explanatory power: zero difference in n • Theory that explains more is accepted • Example: fractal geophysics (chaos theory) versus linear geophysics • Occam’s razor: zero difference in n • Theory that is simple wins against theory that is complicated • Example: Copernicus versus Ptolemy • Predictive power: one n may suffice • Special theory of relativity was proven through one observation of gravitational pull on light during a solar eclipse • Falsifying a theory: one n may suffice • One black swan will disproof the theory that all swans are white (Popper)

10. Large n • Data on natural or human phenomena over time • Can establish historical trends – the more n the better • Modeling of large n through macro-economic or other theories • Mathematical approach to “what if” questions – the more n the better • Natural experimentation; also known as quasi-experimental • Large n is welcome but often difficult to find • Randomized controlled trials • Large n is welcome but costly and difficult to control • Systemic reviews • Sifting through large n to find relevant n

11. Nature of Evidence • The term evidence increasingly refers to outcome of research/studies • Hierarchies of evidence (Campbell collaboration, Maryland hierarchy) focus on large n only • Evidence based on n=1 or no difference in n is no longer recognized as such in some of the literature of the “evidence-movement” • Sciences that use n=1 or no difference in n tend to not be less in policy discussions and provide hardly any countervailing perspectives

12. Causality in Evaluations • Causality in research focuses on new subjects – to proof or disproof causal linkages that are predicted by theory • Causality in evaluations also tackles old subjects and is not focused on proof or disproof of scientific theories, but on what works and why • Interventions take place in a mixed environment of scientific and technical certainties, unproven theories and scientifically unknown territory • Identification of possible causal linkages takes place through a theory based approach

14. From ToC to TonC • A theory based approach may lead to a theory of change identifying causal linkages and assumptions covering these • This may also lead to an identification of what could possibly prevent these causal linkages from “working” • It may also identify what prevents the intervention as a whole to move forward • Analogy: a car needs many working components to function as a car, but take away the wheels and it will stop moving • Identification of these factors leads to a “theory of no change”

15. Meta Analysis • Systemic Reviews go through existing evidence in research and evaluations from the perspective of a specific question • Are cash transfers effective in promoting school attendance? • Many studies and evaluations do not address this question in the exact same way and are thus not accepted as evidence • Health review: only 50 studies accepted from 49.000 • Other forms of meta-evaluations do not pose restrictive questions but pose to explore existing evidence • All quality evaluations on a subject are accepted; and quality evidence in a bad evaluation may also be accepted • Theory based approach

16. Over to Christine Woerlen!