Social Experimentation & Randomized Evaluations

1. Social Experimentation • & • Randomized Evaluations • Hélène Giacobino • Director • J-PAL Europe • DG EMPLOI, Brussells,Nov 2011 World BankBratislawa December 2011 rat

2. The Need for Evaluation • We have little hard evidence on key questions • What is the most cost-effective program to reduce unemployment? What is the real impact of microcredit? • Evidence is important: • for maximizing the impact of limited resources • to give donors and policy makers evidence to select better programs • Evidence provides an objective platform for debate • Evaluations sometimes demonstrate that conventional wisdom needs to be rethought

3. J-PAL: Jameel Poverty Action Lab A network of researchers at universities around the world Focused on randomized evaluations Founded in 2003 by Esther Duflo and Abhijit Banerjee, MIT Professors of Economics

4. Build capacity J-PAL Improve the lives of the poor Evaluate social programs Disseminate the results

5. J-PAL today: 5 Offices & 64 Affiliates

6. J-PAL Evaluations around the World • 306 randomized evaluations in 45 countries • About 150 are completed

7. How to evaluate the impact of an idea? • Implement it on the ground in the form of a real program, and see if it works • To know the impact of a program must be able to answer counterfactual: • How would individual have fared without the program • But can’t observe same individual with and without the program • Need an adequate comparison group • individuals who, except for the fact that they were not beneficiaries of the program, are similar to those who received the program • Common approaches: • Before and after (But many things happen over time?) • Participants vs. Non-participants (But are they different? More motivated? Live in a different region?)

8. Basic set up of a randomized evaluation Target Population Not in evaluation Sample Population Random Assignment Treatment Group Control Group Based on Orr (1999) 8

9. Randomized assignment This method works because of the law of large numbers Both groups (treatment and control) have the same characteristics, except for the program Differences in the outcomes can confidently be attributed to the program

10. The French Context The French context

11. J-PAL Europe • Started in 2008 with 9 affiliates • Today: 16 affiliates (Belgium, Denmark, France, United Kingdom, Sweden) • About 45 projects: • Mainly in developed countries • Mostly ongoing

12. Evaluations in France • France had almost no evaluation culture • French Association of Evaluation • Mostly process evaluations • Endless debates about validity of results • A very limited common knowledge accumulated about the main social problems and how to address them • Urgent need for a method that can provide evidence to direct the public debate

13. The Experimental Fund for Youth In 2008, creation of the Experimental Fund for Youth (Fond d'expérimentation pour la jeunesse) To encourage innovative youth policies To experiment and evaluate these policies $230 million for 2 years, a team of 12 people Within 18 months, 350 projects funded

14. What's New? Government • Encourages initiatives and does not only support its own programs • Decides to support projects only if they are evaluated • Prepares programs for scaled-up conditions from their inception

15. Knowledge against Poverty Chair "Knowledge against Poverty" opens in 2009 at the Collège de France Funded by AgenceFrançaise de Développement for 5 years First Professor: Esther Duflo

16. Fighting Unemployment

17. Many Programs… Counseling the unemployed Counseling and job placement for young graduate job seekers Counseling welfare recipients Discrimination in hiring and anonymous CVs Facilitating youth's access to apprenticeships and encouraging them to complete them Small business training and loans for aspiring entrepreneurs in disadvantaged neighborhoods Supporting 18-25 year-olds through long-term mentoring plus financial assistance

18. Counseling and Job Placement for Young Graduate Job Seekers

19. Policy Issues High unemployment rate of young people Even for young educated people Despite government job placement services Would private operators be able to provide better support? Would strong incentives improve the labor market outcomes? How does this affect employment rates in the overall job market?

20. Context Program launched by French Ministry of Labor in 2007, for 2 years Implemented throughout 10 regions of France Among 57,000 young job seekers Aged between 18 to 30, holding at least a 2 years vocational or university degree No stable work for at least 6 months

21. Description of the Intervention • Intensive counseling and placement provided by private operators • Operators may be for-profit or non-profit • Operators have financial incentives to place the young graduates in a durable job • 10,000 young graduates in the program • Job seekers were randomly selected at the local employment agencies level • The proportion of eligible youth varied from 0%, 25%, 50%, 75% up to 100%, to examine the spillover effect

22. Experimental Design A specific design to identify displacement effect 25% participants 50% participants 0 participant 75% participants 100% participants

23. Take away • Individuals who received intensive counseling were more likely to be employed in the short term • but over the long term (18 months) these differences disappeared • Firms were incentivized to find employment lasting at least 6 months • they focused on jobs which just met that condition • Benefits for the beneficiaries are obtained at the expense of non-beneficiaries • rotation of the unemployed in the queuing process to access employment • may be useful for people having no chance at all?

24. "La Mallette des Parents"

25. La Mallette des Parents (the parents' toolkit) • Is it really possible to improve parents’ involvement ? • Has increased parental involvement any effect on children? • Does the effect on program participants spread out on other families?

26. The Program • Children and their parents in 6th grade in low-income schools • The toolkit: • a booklet with thematic guidelines • 3 meetings with the parents • A DVD in 10 languages • Cost: 1000€ per school (7€/child)

27. Messages to the Parents All parents can help their children Work outside of school is extremely important for success Parents should be involved in their children's homework Children need to feel that their parents understand how school functions and care that they adhere to the demands of teachers and administration

28. Experimental Design: Four Groups Treated Classes (randomized in) Control classes (randomized out) Volonteer Volonteer Compare Non volonteers Non volonteers Compare

29. Parents' Involvement • More interactions with school: • Meetings with teachers • Participation in parents' organizations • Helped more their children at home • Same magnitude as the initial gap in involvement between blue-collar and white-collar families

30. Children's Behavior • Large improvement • even for classmates whose parents were not volunteers • 34% less likely to be sanctioned for disciplinary reasons • Similar results in reduction of absenteeism • Limited but significant impact in test scores in French, no impact in Math

31. Take away • Program demonstrated strong effects on parental involvement and child behavior • to a smaller extent on cognitive achievement • Behavior of all students in the selected classes improved, • including those whose parents did not participate

32. Policy Implication • Simple and inexpensive program • Rigorous evaluation: can convince schools or governments that such action is worth taking • Ongoing generalization in France • New program launched for children in the last class of compulsory school

33. Policy Lessons Directed parent discussion groups are an effective policy tool for increasing parental involvement, even in underprivileged area Increasing parental involvement and awareness of school structure improved student behavior and positively impacted learning Even though only a small fraction of parents choose to participate, the benefits of their involvement were felt by all children in the classroom.

34. More information www.povertyactionlab.org

35. Many thanks !

38. Limits • Not appropriate when: • the impact to measure is a macro impact • the scaling-up of the pilot will modify the impact a lot • the experiment modifies the reality • the beneficiaries are in a context of urgency • It takes a lot of time, sometimes difficult to combine with political needs

39. When to do a randomized evaluation? • As soon as there is an important question and the answer will help fighting poverty • Not too early and not too late… • The program is not too special (can be used in many other contexts) • To be really effective, we need time, talents and .. some money

40. When NOT to do a randomized evaluation When it is too early, the program still needs to be ameliorate If the project is too small (not enough participants) If it was already demonstrate that this program has a positive impact. Use the money for more beneficiaries! If the program has already started

41. The question of ethic Resources are very limited any way This random assignment is usually seen as very fair Every project has to go through an ethic committee J-PAL never works on projects if the cost of the evaluation means less beneficiaries. After the evaluation is over, if necessary, the control group will also get the program

42. Difficulties in the field • We face some resistances: • some people just cannot accept the random assignment perceived as unjust • take up is much lower than previous, • Not always easy to overcome the cultural differences between researchers and partners • Not easy to deal with very decentralized organizations • The evaluation sometimes goes against some financial or political interests Be very careful in choosing the program!

43. Conclusions • Policy needs experimentation… • Too many errors are made all the time; too much time is lost; too much money is wasted • Without experimentation, the stakes are too high. What has been scaled up cannot afford to be wrong. No incentive to learn and progress • Experimentation needs to be serious… • We need to be rigorous in determining success or failure • Otherwise, everyone is free to defend their pet project • Experimentation needs to be creative… • If we are just trying, and accept the possibility of failure, we do not need to think inside the box • This mindset could revolutionize social policy