Using Experiments to Inform P olicy: What have we Learned?

1. Using Experiments to Inform Policy: What have we Learned? Rachel Glennerster Abdul LatifJameel Poverty Action Lab February 13, 2012

2. What have we learned? Many specific lessons about how to improve health, education, empowerment Rigor matters Can look at a wide range of questions with the right measurement tools Flexible tool that can be adapted to fit many constraints logistical, ethical, political Combining theory and RCTs Technical lessons for example on externalities and power Ways to promote experimentation 2

3. Rigor matters: E.g. Microcredit Little rigorous evidence behind many conventional wisdoms Millions went to microfinance with strong claims about benefits Evidence of impact compared women who had and had not taken up microcredit in the same village Randomized evaluations in India, Philippines, Morocco, and Mongolia suggest it is a useful financial tool but benefits less than the claims 3

4. Microcredit results (India) Access to microcredit encourages more small business creation (although it is high to start with). Those with existing business are not hurt by the extra competition Those likely to start a new business cut back on temptation goods and invest more in durables Those unlikely to start a new business consume more (not clear whether this is sustainable) In first 18 months no impact on consumption No impact on women’s decision making, education attendance or health expenditure 4

5. Flexible tool: many ways to randomize Randomized phase in useful when limited capacity to scale up rapidly may rule out collecting long term impacts Randomization at the cut off some well qualified, some not only measures the impact on the marginal, not average person Randomize intensity test of local equilibrium effects Encouragement design when can’t exclude anyone from treatment requires big sample sizes 5

6. Using theory: E.g. fertilizer in Kenya Use of better seed, fertilizers, and irrigation have dramatically improved agricultural productivity in Asia, LatAm but not Africa Series of experiments ask why, looking at fertilizer use in western Kenya (Duflo et al). Big policy debate about whether to subsidize. Fully functioning markets, low take up suggests not profitable found 70% return to inorganic fertilizer Failures in the information market? poor information dissemination farmers who observe the returns increase takeup initially, then tail off suggests behavioral effects 6

7. Using theory: E.g. fertilizer in Kenya II Farmers say they want to use fertilizer, but don’t have the money but fertilizer can be bought in very small packs, and farmers have assets they could sell (that are not making 70% return) Present bias? Offer farmers small time limited discount at harvest time to purchase fertilizer take up increased by 47-70 percent greater take up than with a 50 percent subsidy later in season Sophisticated present bias? when offered a choice some farmers choose the commitment product Commitment devises of this kind successfully used to decrease smoking and increase savings in the Philippines 7

8. The new wave of RCTs Fertilizer exemplifies the new wave of RCTs Could have tested a big government program with many components Instead broke the puzzle down into different components Tested a range of options in a small environment at relatively low cost Understood the problem, designed a solution Now testing a scalable version 8

9. Promoting the use of RCTs Governments need to commit to piloting and rigorously evaluating big new top down reforms huge resources go into these reforms, difficult to unwind afterwards But at same time encourage bottom up innovation and rigorous testing of that innovation. This will generate the new ideas for the next big reforms World Bank, 3ie, French Government, USAID very successful funds to encourage bottom up experimentation Several of these supported by DFID In the UK we have too much national experimentation without proper evidence, and too little rigorous pilot experimentation 9