M&E Systems Strengthening Tool: Recapitulation and Update East Asia and Pacific Regional meeting, 03 to 05 Septembe

1. M&E Systems Strengthening Tool: Recapitulation and UpdateEast Asia and Pacific Regional meeting, 03 to 05 September 2007Bali, Indonesia

2. PART ONE Recap: Background information on the M&E Systems Strengthening Tool

3. GF-centric Exclusively designed and implemented by/for GF Country-driven Self-assessments, generic tools for use by many Focused exclusively onPrincipal Recipient capacities Expanded to integration in the wider M&E Systems General approach Based on open-ended questions Systematic approach Based on a checklist of items Strict assessment tool Broader Assessment, Managementand Planning tools 1- Introduction Evolving paradigms FROM … TO …

4. 1- Introduction Expected Benefits • Guide strategic investments in M&E for grant signature or in existing grants (within the recommended range of 5%-10% of the overall grant budget) • Better identify M&E capacity gapsand corresponding strengthening measures, including through Technical Support • Ensure that investments in M&E are based on identified weaknesses and contribute to the strengthening of the National M&E Systems (avoiding parallel reporting systems) • Enhance the quality of programmatic data to improve program management (and to support Performance-based Funding)

5. … should be implemented for Grant Negotiations • Beginning with Round 6 it replaces the current LFAM&E Assessment Tool for Grant Negotiations; is used as an input to the LFAOverall and Repeat PR Assessment Report • Exception in Rd 6: Implementation as a CP to 2nd disbursement 1 … will be used at anytime during the lifecycle of the Grants based on need or opportunity • Round 1-5: All PRs will be required to implement the M&E Strengthening Tool by the end of 2007 2 1- Introduction Timing of Implementation The M&E Strengthening Tool …

6. 1- Introduction Exemption to Completion of the M&E Strengthening Tool • If a thorough M&E assessment has recently been completed at the country-level, the output of that assessment could be used by the Global Fund. • The scope of that preceding assessment should be similar to the one of the M&E Strengthening Tool • There should be a costed action plan to strengthen M&E systems that the Global Fund could use to guide investments in M&E • The outputs of that preceding M&E assessment (including the action plan) should be communicated to and reviewed by the Global Fund • The PR fills the M&E Systems Strengthening Tool based on the already existing information • Grants in last year of implementation, grants with ‘No Go’ or that have been suspended However

7. 2- Content and structure of the M&E Strengthening Tool The M&E Systems Strengthening Toolassesses 3 areas: M&E Strengthening Tool Purpose 1- M&E Plan • To assess the strength of the Monitoring and Evaluation (M&E) Plan. • To assess the capabilities of the Management Unit (PR) to manage data related to the implementation of the Grants. 2- Data-managementcapacities of the Management Unit (PR) • To assess the strengths of the data-collection and reporting systems per Program Area, including the ability to report valid, accurate and high quality data related to implementation. 3- Data-collection and reporting systems per Program Area (e.g., ARV, DOTS)

8. 3- Implementation Stakeholder Workshop • In general, the M&E Strengthening Tool should be completed through a workshop at the country-level with PRs and relevant stakeholders: • promoting alignment and harmonization of M&E at the country level • enabling leveraging of expertise (notably from Technical Partners) • allowing for a pooling of funding and the identification of shared initiatives • The relevant stakeholders are those that can provide technical input and/or have major investments in the country • Country experiences will vary and contribute to learning of others who are planning to perform the assessment

9. 4- Expected Results Agreed Action Plan with budget to address M&E weaknesses • Strengthening / development of M&E Plan and M&E budget • Informing of wider discussion during Grant negotiations and throughout the lifetime of the Grant • Identification of needs for Technical Support (and of a technical support provider) • Agreement with in-country partners on shared initiatives and funding (e.g., organizing of surveys)

10. PART TWO Implementation status, Partnership and Harmonization

11. Implementation Status in EAP region The tool has been implemented for at least one of the disease components in 32 countries receiving GF grants. 8 of these are in the East Asia and Pacific Region HIV- China, East Timor, Philippines, PNG, Vietnam Malaria- Cambodia, China, Indonesia, Philippines, PNG, Vietnam TB - China, Philippines, PNG, Thailand Preparations ongoing in remaining countries In countries where recent partner efforts to assess M&E systems provide comparable results including costed action plan these could be used to fulfill the requirement 11

12. The tool has been developed under a joint initiative from: • The Global Fund • PEPFAR • USAID • WHO • The Health Metrics Network (HMN) • The Measure EVALUATION Project • Other Partners have joined this initiative, including: • RBM • UNAIDS • World Bank • Stop TB • UNICEF Tool developed in partnership

13. Implementation in Partnership Mobilize Support from technical partners to the M&E assessment process (technical expertise/facilitation) ONE harmonized action plan- use of output information for systems strengthening by all partners Support implementation and follow-up of action plan to ensure sustainability and building of national M&E systems Technical expertise Additional financial resources for strengthening measures beyond GF grant capacity 13

14. Harmonization efforts In order to reduce the burden of frequent and overlapping M&E assessments – Multi-agency Technical Working Group on Harmonization of M&E assessments (Global Fund, World Bank, UNAIDS, Measure EVALUATION and MACRO Intl.) Process underway to refine the tool and come up with one overall M&E assessment tool for HIV

15. APPENDIX I: Components of the Tool –details • The 3 components of the tool enable to comprehensively assess the program’s ability to collect, analyze, use and report reliable M&E data. PART 1: M&E Plan PART 2: Data-management Capacities of the PR PART 3: Program Area 1 PART 3: Program Area 2 PART 3: Program Area 3 PART 3: Program Area 4 Systems for Reporting Numbers of: 1- People Reached 2- Commodities Distributed 3- Individual Trained 4- Service Points Supported Eg. Condom Distribution. Eg. ARV. Eg. PMTCT. Eg. OVC

16. 1- National Strategy and National M&E Plan Is there a National Strategy related to the relevant disease area addressed by the Proposal? Is there a National M&E Plan linked to the National Strategy? 2- Goals and Objectives of the Program Are the goals and objectives of the Program in line with the National Strategy and are they time bound and measurable? 3- M&E Indicators in the Program Do indicators have clear definitions, data sources with baselines and targets? Were indicators selected in collaboration with national/international M&E partners? Is indicator data linked with the National M&E system? 4- Data Dissemination and Transparency Will data from the Program be disseminated properly and will sensitive data remain confidential? 5- M&E Budget Has the Program allocated sufficient financial resources for M&E? APPENDIX I: Components of the Tool –details 1- Assessment of the M&E Plan M&E Strengthening Tool 1- M&E Plan 2- Data-managementcapacities of the Management Unit (PR) 3- Data-collection and reporting systems per Program Area (e.g., ARV, OVC)

17. 2- Oversight of Sub-Reporting Entity Does the Management Unit provide sufficient oversight, guidance and support to SRs? Are reports submitted on time, complete and mistake free? APPENDIX I: Components of the Tool –details 2- Assessment of the Data-management capacities of the Management Unit 1- Data Management Processes and Resources of the Management Unit M&E Strengthening Tool Does the Management Unit possess the resources, procedures, skills and experiences necessary for M&E data management and reporting? Is feedback provided to Sub-Reporting Entities on the quality of their reporting and on program performance? 1- M&E Plan 2-Data-managementcapacities of the Management Unit (PR) 3- Data-collection and reporting systems per Program Area (e.g., ARV, OVC)

18. 1- Systems for Reporting on Numbers of People Reached Do the data collection and reporting systems enable the reporting of valid, accurate and high quality data on numbers of People Reached through the activities of the Grant. 2-Systems for Reporting on Numbers of Commodities Distributed 4-Systems for Reporting on Numbers of Service Points Supported Do the data collection and reporting systems enable the reporting of valid, accurate and high quality data on numbers of Commodities Distributed (e.g., condoms, ITNs) through the activities of the Grant. Do the data collection and reporting systems enable the reporting of valid, accurate and high quality data on numbers of Service Points / Facilities / Organizations Supported through the activities of the Grant. 3- Systems for Reporting on Numbers of Individuals Trained Do the data collection and reporting systems enable the reporting of valid, accurate and high quality data on numbers of Individuals Trained through the activities of the Grant. APPENDIX I: Components of the Tool –details 3- Assessment of the Data-collection and reporting systems per Program Area M&E Strengthening Tool 1- M&E Plan 2-Data-managementcapacities of the Management Unit (PR) 3- Data-collection and reporting systems per Program Area (e.g., ARV, OVC)

19. APPENDIX II: Structure of the Tool –details How does the M&E Strengthening Tool work? Step 1: Diagnosis Step 2: Action Plan Development of a costed action plan: • Identification of Strengths and Weaknesses • Definition of Strengthening Measures with: • Responsibility • Timeline • Funding Implication • Technical support needs Diagnosis through completion of a Checklist: • Answers to Statements/Questions. • Provision of Comments/Explanations.

20. APPENDIX II: Structure of the Tool –details Step 1: Diagnosis List of Statements (describing what should be in place) 5 Answer Categories Yes - Completely Mostly Partly No- Not at All Comments (compulsory if the answer is not "Yes - completely") Review and Verification(to be filled out by the organization reviewing the completed tool)

21. APPENDIX II: Structure of the Tool –details Step 2: Action Plan Strengths and Weaknesses Costed Action Plan with list of Detailed Strengthening Measures (Including Responsibility, Funding and Timeline) Other Relevant Comments Assessment Rating (A-No gaps, B- Minor gaps, C-Major gaps) Review and Verification(to be filled out by the organization reviewing the completed tool)

22. Organization PR, SR, CCM Chair or Vice-Chair According to the country-specific situation Participation National Program for the Disease All PRs - i.e., New, Repeat and Existing PRs (Program Managers and M&E Experts) Relevant SRs (Program Managers and M&E Experts) Government Agencies (MoH, HIS, Central Bureau of Statistics, Policy, etc.); Technical Partners (e.g., WHO, UNAIDS, UNICEF) Other Donors (e.g., World Bank, PEPFAR) And others, as appropriate (NGOs, etc.) Facilitation Funding In-country facilitator chosen by the country (eg., WHO, HMN Focal Points, UNAIDS, etc) M&E Budget from existing Grant Agreements GF Support to i) workshop logistics possible for a number of New PRs and ii) facilitation of workshops APPENDIX III: Organization of a Stakeholder Workshop Duration Between 1 and 3 days - depending on the component assessed as appropriate