Presentation of Final Report The Results of the Thematic Evaluation on Maternal Health (2000-2011)

1. Presentation of Final ReportThe Results of the Thematic Evaluation on Maternal Health (2000-2011) United Nations Population Fund Evaluation Branch Division for Oversight Services New York, 07.11.2012

2. Overview of proceedings Evaluation Branch, Division for Oversight Services

3. Introduction: Objectives, Scope and Methodology of the Thematic Evaluation Evaluation Manager Chief, Evaluation Branch, Division for Oversight Services Evaluation Branch, Division for Oversight Services

4. Objective, users, rationale of maternal health thematic evaluation Evaluation Branch, Division for Oversight Services

5. Thematic & temporal scope MYFF 2000 - 2003 MYFF 2004 - 2007 Strategic Plan 2008 - 11 Relevance of UNFPA support Capacity Development / Human Resources for Health SRH Services (Family Planning / EmONC) Integrating MH into national policies & frameworks MH support in humanitarian contexts Results & evidence orientation Coherence of SRH, Population & Development, Gender Harmonization & coordination of support Coherence between country, regional & global programmes Visibility of UNFPA in maternal health 2010/11 2000 Evaluation Branch, Division for Oversight Services

6. Geographic scope of evaluation 1st stage sampling (“Universe”) 55 programme countries with MMR greater than 300 death / 100,000 life births 2nd sample 22 countries (range of MH performance, GNP, quality of public admin, HIV prevalence) 3rd sample (country case studies) Burkina Faso, Cambodia, DRC, Ethiopia, Ghana, Kenya, Lao PDR, Madagascar, Sudan and Zambia Evaluation Branch, Division for Oversight Services

7. Methodology – Definition of Scope UNFPA results frameworks (MYFFs, Strategic Plan, etc.) MHTF Business Plan, etc. Analysis of strategies of other UN agencies & donors Academic and “gray” literature on reproductive & maternal health ToR / Input from Reference Group Analysis and reconstruction of UNFPA’s own intervention logic (Core fund (& MHTF)) Analysis of the context of UNFPA support to maternal health Main Topics / Issues / Purpose identified in ToR; specified by Reference Group Identification of critical aspects (Core Fund & MHTF) DAC Criteria (Relevance, Effectiveness, Efficiency, Sustainability) 2 sets of Evaluation Questions: Maternal Health Thematic Evaluation (12 EQs) MHTF Mid-Term Evaluation (8 EQs) Evaluation Branch, Division for Oversight Services

8. Methodology –Data Collection • Data were collected to answer 12 Evaluation Question of Thematic Evaluation • Data collected by following means: • Desk review of existing evaluations, reviews and other documents; • Ten country case studies (Burkina Faso, Cambodia, Ethiopia, Ghana, Lao PDR, Madagascar, Sudan (North) and Zambia); focus groups, interviews with UNFPA staff, Government, development partners, civil society, beneficiaries • Online survey of 55 country offices on technical support to country offices & on organizational capacity • Face-to-face and telephone interviews with UNFPA staff members (headquarters, regional and sub-regional offices), other external partners. Evaluation Branch, Division for Oversight Services

9. Methodology –Number of People Interviewed Evaluation Branch, Division for Oversight Services

10. Key Findings and Conclusions: The UNFPA contributions to changes in maternal health outcomes Martin Steinmeyer, team leader (AGEG) Isabelle Cazottes, co-team leader (AGEG) Evaluation Branch, Division for Oversight Services

11. Answers to the Evaluation Questions Selected key findings Evaluation Branch, Division for Oversight Services

12. Evaluation Question 1: Relevance • Resources not allocated proportional to severity of maternal health needs • Resource allocation system has not fully considered relative need levels in each country • Resources distributed based on criteria such as “degree of political support for ICPD”, “absorptive capacity” • No clear, operational definition of maternal health-related vulnerability • Common practice: targeting MH support geographically (regions / districts with high maternal mortality prevalence) • Less common: targeting systemic socio-economic barriers to MH services (cost, distance, transportation) Evaluation Branch, Division for Oversight Services

13. Evaluation Questions 4: Capacity Development – Human Resources for Health • Training support thematic priority of UNFPA • 2000 – 2005: in-service trainings; Challenges: Low alignment with national HRH systems; Little discernible sustained effect on quality of service • 2005 – 2010: pre-service trainings, support of nat. training oversight agencies (curricula review, regulatory frameworks); Advantages: alignment w. nat. structures, sustained effects on SRH training systems • Challenge: Inappropriate deployment & placement of trained staff / high mobility • Linked to weak nat. HRH systems (planning, implementation, monitoring) – beyond SRH / MH • UNFPA offices struggled to approach cross-cutting challenges • Somepartnerships w. HRH offices in health ministries (e.g., Cambodia) Evaluation Branch, Division for Oversight Services

14. Evaluation Question 7 – Emergency Obstetric and Newborn Care (EmONC) • EmONC support prior to 2008, but MHTF has intensified UNFPA EmONC support • Technical assistances, finances sped up EmONC needs assessments; basis for scale-up plans (Lao PDR) • But: Availability of; access to EmONC affected by admin. gaps, cost, weak referral systems • UNFPA offices struggled to approach cross-cutting challenges (barriers) (UNFPA mandate? Capacity?) • Positive exceptions: 1) Analysis of barriers; 2) Long-term Partnerships in- & outside SRH; 3) Advocacy • Burkina Faso: EmONC subsidy, emerged out of long-term UNFPA advocacy, partnership w. Direction de La Famille (MoH), Parliamentarians; increase in facility-based deliveries (28% overall (2003 - 2010)) Evaluation Branch, Division for Oversight Services

15. Overview of key findings for other evaluation questions (selection) Evaluation Branch, Division for Oversight Services

16. Performance patterns of country-level support – Example HRH Health policy arena (& other policy arenas) HRH Policies / Programmes RH policy arena E.g, poor deployment Curriculum reviews; Training Strengthening deployment system CO Management HRH Dep. / MoH SBA plan; midwifery training policy Ohter donors MH Centre / Nursing Councils P&D Gender SRH E.g, WHO Policy advocacy TA SBA SBA assessm. Long-term strategy Evaluation Branch, Division for Oversight Services

17. Performance patterns of country-level support - positive scenario Health policy arena (& other policy arenas) Health Policies / Programmes RH policy arena MH Barriers Implement. Support System Strengthening Management Gov. partner MH policies & programmes Dev. partner Gov. partner P&D Gender SRH Dev. partner Policy advocacy Technical Assistance Data / research Long-term strategy Country Office Evaluation Branch, Division for Oversight Services

18. Performance patterns of country-level support – opportunities for improvement Health policy arena (& other policy arenas) Health Policies / Programmes RH policy arena MH Barriers CO Management Curriculum reviews; Training Gov. partner MH policies & programmes Dev. partner Gov. partner P&D Gender SRH Dev. partner Policy advocacy Technical Assistance Data / research Short-term planning (project by project) Country Office 18 Evaluation Branch, Division for Oversight Services

19. Conclusions

20. Conclusions - Appropriateness of UNFPA maternal health strategy C1: UNFPA maternal health support in programme countries has not been sufficiently based on country-specific medium or long-term strategies • Maternal health support has been more effective when country offices based interventions on multi-annual strategic vision • Current templates not conducive for strategic planning; No requirement to develop multi-annual operational plan. • Country offices without multi-annual perspective more likely to manage interventions separately from each other; • Fewer incentives for staff working on different interventions to coordinate work; pool financial and organizational resources from different funding sources Evaluation Branch, Division for Oversight Services

21. Conclusions - Appropriateness of UNFPA maternal health strategy (2) C2:In its approach to support maternal health, UNFPA has not sufficiently responded to its mandate to focus on addressing the root causes of poor maternal health of the most vulnerable • HQ has not sufficiently defined operational implications of focusing on needs of “most vulnerable” • Country offices did not sufficiently analyse risks and barriers keeping women and girls from accessing MH services (instead targeted geographical pockets of high maternal mortality) • COs did not address weakness of health systems that made specific groups more vulnerable to poor maternal health (e.g., deployment challenges, in particular for rural areas; deficiencies in supervision and service quality control) Evaluation Branch, Division for Oversight Services

22. Conclusions - Appropriateness of UNFPA maternal health strategy (3) C3: UNFPA support of maternal health services at sub-national level has not consistently reflected comparative strengths of UNFPA as primarily knowledge- and evidence-based organization • Significant portion of small SRH budget allocated to sub-national level (sub-national offices, staff, interventions); often one of several organisations active there • Country offices have not consistently used presence at sub-national level to generate data, lessons and knowledge to further national MH policy agenda Evaluation Branch, Division for Oversight Services

23. Conclusions – role & capacity of UNFPA country offices C4:Insufficient staff capacity and skill gaps have negatively affected ability of country offices to act as brokers of maternal health expertise and as facilitator of national and international maternal health commitments and partnerships • Small numbers of RH staff made it difficult to be present in national technical working groups or policy forums • Annual planning based on inadequate templates (AWPs, CPAPs) and limited experience in strategic planning made it difficult to develop multi-annual strategies that combined resources and skills from different areas to facilitate sustained changes in health systems Evaluation Branch, Division for Oversight Services

24. Conclusions – role & capacity of UNFPA country offices (2) C5: Country offices have not received sufficient technical support from regional offices and headquarters to fulfill central role delivering maternal health support • Availability of technical support has been limited in human resources for health, EmONC and midwifery (pre-MHTF), strategic planning, results-based management and monitoring and evaluation • Maternal Health Thematic Fund (MHTF) has been valuable instrument to make available additional resources for technical support Evaluation Branch, Division for Oversight Services

25. Conclusions – partnerships with donors, governments and other stakeholders C6: Varying capacity of country offices to establish partnerships with government, donors and civil society in and outside of reproductive health has reduced ability to address service access barriers and to strengthen MH systems • Long-term, proven working partnerships allow UNFPA to extend reach beyond sexual and reproductive health • Partnerships allow UNFPA to tie maternal health HRH support to larger governmental efforts to strengthen national health systems • Pre-conditions: UNFPA to repeatedly prove worth as reliable partner, gain trust and collaboration, consistent leadership through country representatives Evaluation Branch, Division for Oversight Services

26. Conclusions – Use of Evidence; Monitoring and Evaluation C7: Lack of appropriate monitoring and evaluation mechanisms has affected capacity of UNFPA to assess results of maternal health support and to optimize corporate and country-level MH strategies over time. • Monitoring has focused on activities or higher level societal changes relating to maternal health, • Monitoring system thus not providing information on UNFPA contribution to these changes (which mirrors deficiencies in UNFPA planning) • UNFPA implementing partners w/o required technical capacity to fulfil M&E responsibilities; UNFPA country offices w/o sufficient skilled M&E staff to support implementing partners Evaluation Branch, Division for Oversight Services

27. Conclusions – Added value of the Maternal Health Thematic Fund (MHTF) C8: Maternal Health Thematic Fund (MHTF) has helped to provide much needed financial and staff resources to UNFPA country offices and headquarters in the short and medium-term. • MHTF-financed staff positions bolstered staff capacity of country offices, allowing them to intensify engagement in in EmONC, midwifery, obstetric fistula • Partnerships made available additional technical support (e.g., Averting Maternal Death and Disability (AMDD), International Confederation of Midwives (ICM)) • Helped to raise UNFPA profile and visibility globally & in countries Evaluation Branch, Division for Oversight Services

28. Conclusions – Added value of the Maternal Health Thematic Fund (MHTF) C9: The MHTF has not been sufficiently integrated into organizational structure of UNFPA and the overall planning process at country level to ensure sustainability of its interventions • MHTF has proved instrumental in facilitating EmONC assessments, midwifery needs assessments, which constituted basis for governments to launch MH-relevant reforms • MHTF not sufficiently integrated into UNFPA planning to ensure timely resource mobilization from within UNFPA and partners for continued support of reforms Evaluation Branch, Division for Oversight Services

29. Questions? Comments? Evaluation Branch, Division for Oversight Services

30. Recommendations

31. Recommendations: Reviewing UNFPA maternal health strategy and support R1:UNFPA should revise internal procedures, tools and templates for strategic planning. New approach should require country offices to develop maternal health support strategies for medium to long-term, and to detail how resources from different sub-programmes will be used to implement these strategies. (based on conclusions C1, C7) • Present detailed analyses of specific political, administrative, socio-economic challenges in revised CPDs and CPAPs and corresponding strategy to address them • Put in place multi-annual plans that detail theory of change, risks & assumptions and monitoring indicators for strategy Evaluation Branch, Division for Oversight Services

32. Recommendations: Reviewing UNFPA maternal health strategy and support (2) R2:UNFPA needs to better define operational implications of targeting the needs of the “most vulnerable”. Concept is relevant part of maternal health strategy, yet it is too vague in its current form to guide maternal health programming at country level. (based on conclusion C2) • Develop typology of MH-relevant barriers for different types of services and contexts • Prepare operational and programming guide to a) offer clear definition of MH vulnerability and the contributing social, economic, political variables; b) identify policy sectors that influence MH vulnerability; c) lay out options to influence determinants of vulnerability; d) define UNFPA role in contributing to donor response; e) discuss use of partnerships Evaluation Branch, Division for Oversight Services

33. Recommendations: Reviewing UNFPA maternal health strategy and support R3: UNFPA needs to increase focus on knowledge generation and learning, to make use of lessons from sub-national service delivery support (and other interventions) to inform evidence-based policy advocacy at core of UNFPA mandate. (based on conclusion C3, C7) • Service-delivery support needs to contribute to body of knowledge on maternal health support in programme country and beyond • Strengthen provisions for monitoring, evaluation, learning, in particular for interventions at sub-national level Evaluation Branch, Division for Oversight Services

34. Recommendation: Improving capacity of country offices R4:UNFPA needs to better align capacity and skill mix of staff and managers to ensure that country offices can fulfill their role as knowledge brokers and facilitators of evidence-based approaches to improve maternal health. (based on conclusions C1, C4, C7) • Country offices to develop resource plan (part of CPAP) to allocate staff time to strategy components • Strengthen skills related to health policy and management, public health ; also project management, M&E • Strengthen accountability of country representative for strategic positioning of country offices for partnerships, advocacy to complement UNFPA interventions Evaluation Branch, Division for Oversight Services

35. Recommendation: Improving capacity of country offices R5: Planning process for regional-level technical support to be better aligned with long-term strategic and operational planning for maternal health support at country level. (based on conclusions C4, C5) • Long-term strategic planning at country level needs to be mirrored by long-term planning of technical support at regional level • Regional offices’ planning processes need to address current country-level programming needs and anticipate future support requirements • RPAPs / Country Programmes to be developed jointly with country offices, HQ, MHTF, GPRHCS, including resource plan (similar to country level) Evaluation Branch, Division for Oversight Services

36. Recommendations: Improving guidance on UNFPA partnerships R6:Anchor concept of partnerships more firmly in strategic documents, operational guidelines and job descriptions of managers. Strategic documents need to explain importance of different types partnerships to ensure sustainable results. UNFPA managers need to be responsible for setting up required partnerships at country, regional and global levels. (based on conclusions C1, C6) • Develop typology of stakeholders and their significance for addressing root causes of poor maternal health • Strengthen capacity of regional offices to support country offices in establishing partnerships • Job descriptions of managers need to emphasize their responsibility for strategic positioning of UNFPA through partnerships (performance criterion) Evaluation Branch, Division for Oversight Services

37. Recommendations: Improving UNFPA provisions for monitoring R7: Strengthen result-oriented monitoring for country offices to measure results and not only activities and inputs. Assign greater responsibilities and offer more guidance to country offices for supporting the set-up of appropriate monitoring mechanisms with implementing partners. (based on conclusions C1, C4, C5, C7) • Develop guidelines for results-oriented monitoring & related training resources • Strengthen responsibilities and capacity of country offices to support set-up of monitoring mechanisms with implementing partners (consider partnerships) Evaluation Branch, Division for Oversight Services

38. Recommendations: Revise role & integration of MHTF R8: Strengthen MHTF as catalytic tool to facilitate implementation of evidence-based maternal health interventions. Clarify that MHTF is not only responsible for launching initiatives (EmONC, midwifery), but also for assisting country offices to support these initiatives until completion. (based on conclusions C8, C9 + MHTF-mid-term evaluation) • Clarify: MHTF helps launch initiatives; and supports their implementation (business plan) • Update guidance to country office on purpose and role of MHTF • Support to expect from MHTF • Responsibilities of country offices when working with MHTF funds • Harmonize MHTF planning with (proposed) multi-annual planning approach for core funds Evaluation Branch, Division for Oversight Services

39. Recommendations: Revise role & integration of MHTF R9: UNFPA should use MHTF funds to carry out pilot interventions on selected core maternal health issues, such as the development of appropriate support strategies to better target populations with high vulnerability to poor maternal health. (based on conclusions C3, C8, C9) • Inventory of MH issues and topics that require pilot-testing • Invite country offices to submit proposals for pilots for maternal health interventions • Ensure access of country offices to required resources for pilots through MHTF • Ensure analysis and dissemination of results with MHTF resources Evaluation Branch, Division for Oversight Services

40. Questions? Comments? Evaluation Branch, Division for Oversight Services

41. NEXT STEPS Evaluation Branch, Division for Oversight Services

42. Maternal Health Thematic Evaluation Report and related products are available at the following link: http://www.unfpa.org/public/home/ about/Evaluation/EBIER/TE/pid/10094 Evaluation Branch, Division for Oversight Services

43. Anyotherquestions? Contactus: evb@unfpa.org Thank you for your attention Evaluation Branch, Division for Oversight Services