Alice Mesnard EdePo , IFS

1. The evaluation of the Familias en acción programme in Colombia:do conditional subsidies improve education, health and nutrition outcomes? Alice Mesnard EdePo , IFS

2. Outline • the Familias en acciónProgram • how do we evaluate its effects ? Methodological issues and data 3. Results Education, health and nutrition in poor households 4.What can we conclude ? significant effects on development of children in Colombia Is it enough to assess the effectiveness of the program?

3. Familias en acción • Familias en Accion is a Conditional Cash Transfer program. Its final aim is to foster the accumulation of human capital. Why is it important ? • CCT provides monetary incentives for families to invest in human capital (education and health) • Conditional cash transfers are now being implemented in: • Mexico, Honduras, Nicaragua, Panama, Brazil, Argentina, Colombia, Turkey, Bangladesh • In Colombia, the program is targeted towards the 20% poorest households living in towns with less than 100k people, with enough education and health infrastructure, and with a bank

4. Within each villages 20 % poorest hhds how? indicator summarises living conditions (has a toilet, running water, quality of walls, income..) Create Sisben levels 1 to 6 : take only Sisben 1 hhds = poorest of the poor Need to be registered in the village in dec 1999 • Infrastructure : villages with banks otherwise transferring cash is dangerous , complicated Schools : aim is to evaluate enrolment in schools. Assumes that schools are supplied

5. Familias en acción • The program is made of two components: Education and Health • Education: • Eligible families with children aged 6-17 receive a subsidy per child conditional on school attendance. • The subsidy is about 10US$ per month for primary school children and 20US$ for secondary school children • Nutrition and health: • Eligible families with children aged 0-5 receive about 30US$ per month conditional on registering the children on growth & development checkups • Mothers are also ‘encouraged’ to attend some ‘talks’ with health professionals • All subsidies are given to the mothers

6. Political context In 2002 the programme started, financed by World Bank, Inter American Bank for Development and the Colombian govnt Pgm is viewed as a replacement of Hogares Communitares, which has been providing free child care and subsidised food to children of all households since 20 years The loan is running until next year : if gvt wants to continue it has to refinance it alone: how ? Maybe by cutting pre-existing program He needs to evaluate the FA programme

7. Main features of the beneficiary families • Average family size: 7 • Average monthly consumption 150US$ • Including consumption in kind • Food consumption accounts for about 60% of consumption • 85% of households report consumption in kind • Under civil war

8. Treatment municipalities 800 700 600 500 400 300 200 100 0 0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100 Age structure

9. Focus on children development Children 0 to 6 years old • Chronically malnourished: 23.7% • Diarrhoea in 15 days: 17.1% • Symptom of respiratory disease: 44% • Compliance Growth and Dev checkups: 30%

10. School enrolment by age and type of town 100,00% 80,00% 60,00% 40,00% 20,00% 0,00% 7 a 9 años 10 a 11 12 a 15 16 a 17 control con pago Tratamiento con pago Control sin pago Tratamiento sin pago School enrolment

11. 2. The evaluation of Familias en Accion • The evaluation will be based on the comparison of outcomes of interest between individuals living in treatment towns and those living in control towns these outcomes are : enrolment in schools, food consumption, anthropometrics, self assessed symptoms, compliance with G & D check ups • Idea of an experimental set-up

12. Methods: Potential outcomes model

13. Why is it difficult to measure ? • We do not have the counterfactual Treatment municipalities have received the program • Can we consider instead the outcome in control municipalities ? It is important to consider the existence of pre-existence differences between treatment and control municipalities. Differences in outcomes reflect pre-existing differences or effect of different variables

14. Methodology in a nutshell To take into account pre-existing differences and differences in other variables we use a combination of two standard evaluation techniques: Method 1 : Selection on observables (parametric and non parametric) Method 2 : Difference in difference effect of the program= differences in school enrolment between treated and non treated children minus pre-program differences We need a baseline database and a follow up one

15. The data • Sample: 11,500 households in 67 treatment and 62 control municipalities • Baseline collected between June and November 2002 • First follow up collected between July and December 2003 • Complete household survey. Info about all household members. Household survey is quite long: about three hours and a half • Questionnaires to household heads, mothers etc.... • Survey to major, health care centres, schools, nurseries • Good quality of data : attrition =6%

16. Method 1 Selection on observables • Suppose: • Y=b X + c D + u Where X are some observable variables, u is an unobserved random variable and D=1 indicates that the program is on (so that is 1 in treatment towns and 0 in non treatment towns). • Assume that u is orthogonal to D conditional on X. • This implies that after controlling for observable household and town characteristics the program is assigned randomly • What is the effect of the program ? • In this case, we can estimate the effect of the program (c) by pooling treatment and control households and estimating the equation above.

17. Selection on observables (continued) • We can even generalize the technique above to take into account in a flexible way the effect of X and D • The general idea is to compare treated households to control households that are similar in the X’s . • Generalisation if there are several variables : summarised by a score of been treated Let Prob{D=1 | X} = p(X) p(X) is called the propensity score We compare the outcomes between households with close propensity scores. This is called the Propensity score matching method

18. Variables included in the Propensity Score (to match) Head and spouse, household composition (age, education, height, number of brother and sisters, age composition) Household infrastructure( water by pipe, gas by pipe, connection to sewage, rubbish collection, telephone, house walls materials, house ownership, household has suffered shocks related to illness, death or violence in the last three years) Municipality level variables (region of the country, altitude, index of quality of life of the municipality, population size, number of schools, number of students per teacher, number of square meters in the school per student, presence of public hospital in the municipality, number of health care centres, number of small health clinics, number of pharmacies, percentage of households with water and sewage by pipe, occurrence of strikes/taskforce desertion in the health care providers of the municipality)

19. Method 2 Difference in difference • The 2 methods above assume that conditional on the X’s treatment and control villages are statistically identical. • Of course it is possible that there is some unobserved characteristic of the town that is related to the program and will confuse the effect. i.e. Y=b X + c D + v+u with v correlated to D ? • If such characteristics v are constant over time and we have data pre and post program, we can use diff in diff methods The effect of the program = the post program average differences in school enrolment between the treatment and control municipalities minus the pre program differences

20. Difference in difference effect Y treatment effect Control t=1 t=0

21. Example : diff and diff effect • Before the program, school enrolment rates In treatment towns : 60% In control towns : 50% • After the program, in treatment towns : 80% in control towns : 60% What is the effect of the program ?

22. Answer 10 % Diff and diff effect = (80% - 60%)-(60%- 50%)

23. 3 Evaluation Results • In most cases, we look separately for urban and rural parts of municipalities • More important results • School attendance • Household consumption • Nutrition • Health • Other results : child labour , migration See the report on the IFS website www.ifs.org.uk/edepo/

24. Table 1. Impact of FA on percentage of children who attend school * Statistically significantly different from zero at the 95% confidence level.

25. Impact on school attendance • Higher impacts in rural parts • Higher impacts on older children in rural parts • There are also positive impacts on older children living in urban areas • Unsignificant effects on young children. if the aim is to increase enrolment in primary school : wasted money . redesigning pgm ? • Other issue : nothing on quality of school ? We know repeated school • Child labour : effects in terms of number of hours Probably school and work are not perfect substitutes

26. Table 2. Impact of FA on total consumption and on food consumption (in Colombian pesos) * Statistically significantly different from zero at the 95% confidence level

27. Impact on consumption • Household consumption increases in about 15% most of it due to food (+20%) : not very interesting : poor people increase consumption if their income increase • More interesting results on food components: The ones that increase more are protein rich foods (meat, chicken, milk and eggs). • Apart from food, there are increases in: Clothes and shoes for children (12,000 pesos) Education (in the urban area) (8,000 pesos) No effect on alcohol, tobacco, or adult clothes. Why is it important ? Pb of answers ? • Very interesting results : FA improves nutritional outcomes

28. Table 3. Impact of FA on selected components of household consumption (in Colombian pesos)

29. Impact on number of days per week that children had… *, 10% **, 5%

30. Table 4. Impact of FA on percentage of children with up-to-date schedule of preventive healthcare visits * Statistically significantly different from zero at the 95% confidence level.

31. Health Impacts Significant effects Probability that children adequately complied the growth and monitoring visits according to the health ministry guideline increased by 23 % for below 2 years old, and 32% for 2 to 4 years old. Probability that children suffered diarrhea in the rural part decreased by 10% in the rural part

32. Table 5. Impact of FA on percentage of children who suffered from diarrhoea in the 15 days prior to the interview

33. Table 6. Impact of FA on boys’ heights (in centimetres)

34. Effect on height • 1/2 cm effect on young children only • What about older children ? Effects take more than a year to be significant • Hope to evaluate longer term effects with next follow ups (3 years effect)

35. 4 Conclusions: Is the program effective ? • FA improves school enrolment among older children • FA improves positively the nutritional status of younger children, and the occurrence of diarrhoea in the rural part of municipalities • FA has a very large effect on the attendance to preventive health care visits

36. other issues ? • Cost benefit analysis • Comparison with other programs supply of schools, training of teacher etc… Which ones are more effective ? How can we compare their results ? If we have similar data on children attending both programs ? ex : Hogares comunitarios is a nutrition/child care program introduced all over Colombia in the early 1980s FA widely perceived as a substitute for Hogares • Targeting issues : waste of money • Political economy : whom do you want to help ?

37. example : FA & Hogares comunitarios: which one? • Hogares comunitarios is a nutrition/child care program introduced all over Colombia in 1986. • The program is targeted to poor people (sisben 1,2,3) • Subsidized child care: • Children 0-6 can attend for a small fee • Receive food and child care from a ‘community mother’ • FA widely perceived as a substitute for Hogares Hhds have to choose between two pgms

38. Percentage of children attending Hogares Comunitarios

39. Hogares Comunitarios & FA • Evaluation performed in the ‘control towns’ • Different methodology as the program is universal: • Compare participants to non participants • But take into account that participation might be related to the outcome of interest • Use distance as an ‘instrument’ • Startling results: • HC has improves height of children 0-5 by about 2 cms. • It has also long run effects on school achievement • It increase considerably employment of mothers • Are Familias and Hogares really substitute?

40. Interactions between HC and FA • We know that HC benefit mostly those children older than 24 months • and FA seems to benefit most those children younger than 12 months. • HC and FA seem to be complements rather than substitutes