Microfinance and Home Improvement: Using Retrospective Panel Data to Measure Program Effects on Discrete Events

1. Microfinance and Home Improvement: Using Retrospective Panel Data to Measure Program Effects on Discrete Events Bruce Wydick Professor of Economics, University of San Francisco Visiting Professor , UC Santa Barbara joint with Craig McIntosh University of California at San Diego Gonzalo Villaran University of San Francisco

2. Background: Microfinance Summit: As of January 2006, 3,133 microcredit institutions have reported reaching 113,261,390 clients, 81,949,036 of whom were among the poorest when they took their first loan. Still amazing that don’t have robust results of positive microfinance impact (Armendáriz de Aghion and Morduch, 2005). Recent renewed emphasis on program impact appraisal (e.g. Easterly, 2006; Center for Global Development, 2006)

3. An “evaluation gap” has emerged because governments, official donors, and other funders do not demand or produce enough impact evaluations and because those that are conducted are often methodologically flawed.” --Center for Global Development (Saveduff, Levine, Birdsall, 2006 with E. Duflo, P. Gertler, etc.)

4. Problems with lack of quality impact studies: 1. Accurately measuring program impacts has historically been logistically difficult, time consuming, and costly.

5. Problems with lack of quality impact studies: 1. Accurately measuring program impacts has historically been logistically difficult, time consuming, and costly. 2. Many institutions would like to evaluate the effectiveness of their programs ex-post to implementation, creating problems with the establishment of baseline surveys, control groups, and other means of identification.

6. 3. Use of instruments or program rules (e.g. Pitt and Khandker, 1998) to obtain program impacts is theoretically appealing, but practically problematic: If available, instrumental variables will differ from one situation to the next. Finding instruments in a particular context strongly correlated with program access, but uncorrelated with impact variables, requires substantial ingenuity. Point: complicates use of a standardized instrumental variable approach.

7. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.)

8. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.) Gomez and Santor (2003) use statistical matching model to identify the effect of group lending relative to individual lending among 1389 individual and group borrowers among 1,389 borrowers in Canadian lending institution.

9. 4. Matching Models -- creating artificial controls in order to identify treatment effects. (e.g. propensity scores, nearest neighbor, etc.) Gomez and Santor (2003) use statistical matching model to identify the effect of group lending relative to individual lending among 1389 individual and group borrowers among 1,389 borrowers in Canadian lending institution. Problem: Cannot control for unobservables.

10. 5. Randomized experiments--become very popular as way of ascertaining impact of development programs. Maximum degree of exogeneity in treatment and control, allowing means of overcoming self-selection, endogeneity, and omitted variable bias (common to microfinance) Most elegant way of ascertaining impacts, and least controversial.

11. Difficulties: a) To create control group needed for identification of treatment, necessary that treatment withheld for some who desire it so impact can be measured on treatment group relative to the control…often undesirable or infeasible

12. Difficulties: a) To create control group needed for identification of treatment, necessary that treatment withheld for some who desire it so impact can be measured on treatment group relative to the control…often undesirable or infeasible b) Some treatments (e.g. microfinance) may take years to realize full effects on household--timeframe may not intersect with time one can “hold off” control group…"bleeding" of control group.

13. Difficulties: a) To create control group needed for identification of treatment, necessary that treatment withheld for some who desire it so impact can be measured on treatment group relative to the control…often undesirable or infeasible b) Some treatments (e.g. microfinance) may take years to realize full effects on household--timeframe may not intersect with time one can “hold off” control group…"bleeding" of control group. c) To avoid bleeding of control group, often short-term, but then only capture effects of initial adopters.

14. d) Point estimates from randomized experiments are subject to influence of time-specific economic shocks occurring within the relatively narrow timeframe of the experiment…understates true standard errors. e) Because randomized field experiments typically represent a snapshot of program impact over a short time frame, they are often unable to capture important dynamics of treatment impact. Ideally, we would like to understand how an intervention affects a treatment group over time.

15. Our paper presents a methodology for ascertaining welfare changes brought about by development programs that may be applicable in a variety of contexts (explain later). Main Advantages: Uses a single wave of cross-sectional surveying. Impact evaluation can be undertaken ex-post. No firm requirement for standard control group. Allows for a dynamic analysis of impacts

16. Our methodology appropriate when… Program has existed for a number of years. Has been phased in over time in different geographical regions or identifiably separate populations for reasons that are independent of dependent variables. Stable populations with little geographical movement.

17. Methodology uses a single cross-sectional survey to create a retrospective panel data set based on discrete, memorable events in the history of households. e.g. install indoor plumbing, new house, purchase of first cell phone, miscarriages, infant deaths, land purchases etc. Identification of impact rests in analyzing the timing of these events with respect to the timing of treatment. Test is for differences in the probability of these major events within window surrounding the treatment. (Post vs. Pre--under conditions of exogeneity)

18. We apply methodology to studying the effects of a microfinance program in rural Guatemala on home improvements

19. We apply methodology to studying the effects of a microfinance program in rural Guatemala on home improvements Study discrete changes in the probability of major dwelling improvements, upgrades of walls, roofs, floors, the installation of indoor toilets, and the purchase of new land + compilation of these.

20. We apply methodology to studying the effects of a microfinance program in rural Guatemala on home improvements Study discrete changes in the probability of major dwelling improvements, upgrades of walls, roofs, floors, the installation of indoor toilets, and the purchase of new land + compilation of these. Use linear probability estimator that incorporates household and year fixed-effects. (Chamberlin, 1980)

21. Sneak Preview of Results: Microfinance loans for enterprise expansion is likely to exhibit significant, positive effects on some dwelling upgrades, especially to walls & floors. Roofs uncertain. Apparently not for toilets and land.

22. Identification of impacts is achieved through the existence of counterfactual, i.e. what would have happened to treatment group in the absence of a particular treatment. Counterfactual in a randomized field experiment in microfinance is the resulting level or change in impact variables realized within a subset of borrowers in the control group who desired credit but were prevented by researchers from receiving it.

23. Counterfactual that yields identification in our methodology is difference in probability of discrete events among those who received the treatment in separate years, controlling for these differences in years with village- and year-level fixed-effects.

24. Main contributions: 1. Offer a sequence of steps that include diagnostics on the data to check for supply-side & demand-side endogeneities in the rollout of a program. 2. Establish framework for thinking about when and how retrospective panel data can be used in impact analysis. 3. When data meets certain diagnostic criteria, allows us to examine the dynamics over probability of these discrete events before and after treatment & test for significance of a type of treatment effect.

25. Steps involved in methodology: Part A: Survey

26. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations.

27. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods.

28. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel.

29. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality

30. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality E.g. smallpox vaccine → lower instances of smallpox

31. Steps involved in methodology: Part A: Survey Step S1: Identify a program that has been phased in over a number of years in different geographical areas or among different populations. Step S2: Carry out random survey of program participants who have been given access to the treatment in different time periods. Step S3: Identify discrete historical changes with a theoretical basis for causality from the treatment & create historical panel. E.g. fresh water → reduced infant mortality E.g. smallpox vaccine → lower instances of smallpox E.g. microcredit access → higher enterprise profits → more rapid home improvements

32. Steps involved in methodology: Part B: Econometrics

33. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program.

34. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect

35. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity

36. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity Step E4: Estimation of the Take-up Effect

37. Steps involved in methodology: Part B: Econometrics Step E1: Check for supply-side endogeneity in the rollout of a program. Step E2: Estimation of the Retrospective Intention to Treat Effect Step E3: Testing for Demand-Side Endogeneity Step E4: Estimation of the Take-up Effect Step E5: Treatment Window Regression and F-test of Take-up Effects

38. 2005 (BASIS/USAID funded) survey of 218 households located in 14 different villages near Quetzaltenango, Guatemala. MFI: Fe y Alegria: 3,000 new clients/year Questionnaire ascertained changes in different categories of dwelling improvement: upgrades to walls, roofs, floors, plumbing, and increases in land. Each borrower was asked about changes in these variables during the history of the household, and the timing of these changes.