AN IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR GENERAL HEALTH SERVICES IN RWANDA

1. Project team Agnes Soucat Miriam Schneidman + team AN IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR GENERAL HEALTH SERVICES IN RWANDA Evaluation team: Paulin Basinga Paul Gertler Jennifer Sturdy Christel Vermeersch Damien de Walque

2. Presentation plan Country and program information Impact evaluation design Descriptive results from the baseline study

3. SECTION 1: COUNTRY AND PROGRAM INFORMATION

4. RWANDA general & health sector Total population : 9,038,001 (2005) 30 Administrative districts GDP per capita of $230 33 District Hosp. and 369 Health centers HDI: ranked 158th (2004) MMR: 750 per 100,000 (DHS 2005) IMR: 86 per 1,000 (DHS 2005) HIV: 3.1% (DHS 2005)

5. Relevance and Severity of the Health Issue Addressed Diagnostic Shortage of human resources for health services Low levels of productivity and motivation High levels of absenteeism Low user satisfaction & poor quality of service lead to low use. Increase morbidity and mortality Goal Increase number of trained medical personnel Increase motivation Improve quality of services Increase personnel income Policy Response Performance Based Contracting & Financing

6. National PBC model for Health Centers 16 Primary Health Care indicators, eg: New Curative Consultation = $0.27 Delivery at the HC = $3.63 Completely vaccinated child = $ 1.82 14 HIV/AIDS indicators, eg: One Pregnant woman tested (PMTCT) = $1.10 HIV+ women treated with NVP = $1.10 Separation of Functions between stakeholders

7. Fig: Implementing organizations

8. SECTION 2 : IMPACT EVALUATION DESIGN

9. Evaluation Design • Make use of expansion of PBC schemes over time • The rollout takes place at the District level • Treatment and control facilities were allocated as follow: • Identify districts without PBC in health centers in 2005 • Group the districts based on characteristics: • rainfall • population density • livelihoods • Flip a coin to assign districts to treatment and control groups.

11. Roll-out plan • Phase 0 districts (white) are those districts in which PBF was piloted • NOT part of the impact evaluation • Phase 1 districts(yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’ • Phase 2 districts (green) are districts in which PBF is phased in later; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. PBF is being introduced in these districts in 2008.

12. Program Implementation Timeline

13. Quality assurance in comparisons • Law of large numbers does not apply here… • Proposed solution: • Propensity scores matching of communities in treatment and comparison based on observable characteristics • Over-sample “similar” communities in Phase I & Phase II • It turned out • Couldn’t find enough characteristics to predict assignment to Phase I • Took a leap of faith and did simple stratified sampling

14. More money vs. More incentives • Incentive based payments increase the total amount of money available for health center, which can also affect services • Phase II area receive equivalent amounts of transfers • average of what Phase I receives • Not linked to production of services • Money to be allocated by the health center • Preliminary finding: most of it goes to salaries

15. Surveys • General Health facility survey (168 centers) • General Health household survey • HIV/AIDS facility survey (64 centers) • HIV/AIDS household survey Note: HIV/AIDS study is for another presentation

16. Design and sample

17. General Health Household Survey: content • Socio-economic information • Anemia finger prick test: children 12-71 months old • Malaria dip stick test: children under age 6 • Anthropometrics: <6 years old • Mental health: mothers, pregnant women, adults over age 20 • Sexual history and preventative behavior knowledge • Pre-natal care utilization and results • Parents or caretakers were asked for information regarding child (<5 years) health status

18. Health Centers Survey • All 168 centers in General Health; 64 in HIV/AIDS • General characteristics • Human resources module: Skills, experience and motivations of the staff • Services and pricing • Equipment and resources • Vignettes: Pre-natal care, child care, adult care VCT, PMTCT, AIDS detection services • Exit interviews: Pre-natal care, child care, adult care, VCT, PMTCT

19. Analysis Plan • All analyses will be clustered at the district level • Compare the average outcomes of facilities and individuals in the treatment group to those in the control group 24 months after the intervention began. • Use of multivariate regression (or non-parametric matching) : control confounding factors • Test for differential individual impacts by: • Gender, poverty level • Parental background (If infant : maternal education, HH wealth)

20. SECTION 3: DESCRIPTIVE RESULTS FROM BASELINE SURVEY

21. Health Facilities Baseline Survey

22. Classification of facilities

23. Financial Capital Resources Average Annual Funding at Facility Level 60.00% 50.00% RWF Major Source RWF 2,416,597.00 Second Source 2,069,715.00 40.00% Third Source 30.00% RWF 727,169.90 20.00% RWF 295,745.90 RWF 297.47 10.00% Health Religious Consultation Drug Fees Lab Fees 0.00% District Fees Other User Fees Drug Sales Religious Other Donor Government Main Sources of Funding Main Sources of Funding at Facility Level

24. Human resources On average: 1 doctor for every 31,190 individuals, 1 nurse for every 4,835 individuals

25. Availability of laboratory tests

26. Availability of equipment

27. Availability of drugs

28. 8% Family history of genetic problems 13% Medicaments that is taking right now 17% 18% Lost/gain of weight, nausea, vomiting 21% 21% 85% Late menstrual dates 80% Number of previous pregnancies 85% 86% Number of previous miscarriages 64% 92% HealthProvider Patient Prenatal Care procedures:Comparison between health provider & patient responses

29. 93% Listen featal heatbeat 86% 61% Blood presure measured 81% 97% Abdomen examination 81% 17% Height measured 72% 19% Pelvicexamination 58% 83% Weight measured 52% Health Provider Patient Prenatal care Physical ExaminationComparison between health provider and patient responses

30. Quality index: Descriptive statistics 31

31. PATIENT SATISFACTION Negative significant correlation between satisfaction and: The length of wait time Total time spent attending the facility, The cost of the visit The cost of the medications. No difference between satisfaction of mutuelle members and non-members But mutuelle membership is significantly positively correlated with cost Prenatal visit: positive correlation with the cleanliness of the facility Child care: lower levels of satisfaction in Waiting time, Total time spent for visit, Cleanliness of the facility, Privacy during the exam, Attitude of the staff, Explanation of the child’s condition during the exam

32. Household Baseline Survey

33. Sample description • 2159 HH, 10,880 individuals • Gender balance (49,3% male and 50,7 female) • Average HH size is 5.71 individuals, • Age range of <12 months to 96 years old. • Half of the sample is below 12 years old • 75% of the sample is under the age of 30 years old. (Sampling strategy)

34. HH Education by phase

35. Household assets 36

36. Maternal health service utilization

37. Maternal health service utilization

38. Immunization

39. Conclusion • HH Results comparables to the recent DHS • Validity of treatment – control groups • Of 110 key characteristics and output variables of HF, the sample is balanced on 104 of the indicators. • Of 80 key HH output variables, the sample is balanced on 73 of the variables. • Majority of the indicators which differ between Phase I and Phase II are results from patient exit interview