Joint Evaluation of the Paris Declaration, Phase 2

1. Joint Evaluation of the Paris Declaration, Phase 2 Evaluation Framework & Workplan Presentation to International Reference Group 30 Nov 09

2. Umbrella Framework for All Elements in Phase 2 • Overview • Evaluation Profile: Reasons for the Evaluation, Objectives, Audiences and Stakeholders, Approach • Approaches to Methodology: Basis, Principles, The Evaluation Framework and Core Questions, Core components of the methodology, Anticipated methods, Ensuring comparability, Ensuring validity and reliability. • Accountabilities and Responsibilities • Workplan and Schedule • The Overarching Evaluation Matrix for Phase 2 • Draft Generic TORs for Country Evaluations • Draft Generic TORs for Donor/Agency HQ Evaluations • First Draft Outline for Evaluation Synthesis Report • Two framework diagrams from preparatory studies

3. Building blocks of the synthesis SYNTHESIS • EVALUATION QUESTIONS • 4. Alternatives • 3. Development outcomes • 2. Process and intermediate outcomes • 1. Context DONOR STUDIES COUNTRY STUDIES SUPPLEMENTARY STUDIES PHASE 1 RESULTS

4. Relationship to ToRs • The Framework is the umbrella that sets out all substantive elements of the Evaluation, its governance & broad approaches to methodology; • Generic ToRs for Country and Donor/Agency HQ evaluations will operationalize the main components, with direct guidance to teams; • The whole structure and synthesis is to be based on a single, clear set of evaluation questions and a simple but rigorous approach to ensure comparative findings that are credible, relevant and useable by its key audiences.

5. Overall evaluation approach • Accepts special methodological challenges; • Process and results designed for accountability (Seoul & at home) and learning (to strengthen policies & practices); • Fully joint, relying primarily in Phase 2 on the 20+ country evaluations, including donors on the ground ; • Seven donor/agency HQ-level studies, plus possible updating of Phase I. A small number of “supplementary studies”. • Evaluating a) to what extent the PD has been implemented, and b) in so far as it has been implemented, what the results have been in terms of aid effectiveness and contribution to development results; • Focus on the workings of country-donor partnerships and their development outcomes at country-level; • Implementation or “process;” and • “Outcomes” or “Results” in terms of aid effectiveness and contributions to development results.

6. Approaches to Evaluation Methodology • Summative and formative; • A clear, simple and straightforward common approach, language, and methods for the evaluations; • Apply existing data sources, avoid duplicative and unnecessary demands; • Take account of early preparatory work on the many complex factors and relationships at work (see diagrams in Appendix E). Matrix to Mgt. Group in Sept/Oct 2009.

7. The Evaluation Framework and Core Questions • A manageable set of agreed common Core Evaluation Questions and sub-questions, with common methods, for robust comparative findings; • Focus on the most important, results-oriented questions of most interest to most users; • If needed, partner country and Donor/Agency HQ evaluations to supplement their answers to the common Core Questions with questions of special interest/priority to them; • Allow for the integration of Donor/Agency HQ Evaluations; key cross-cutting assessments, e.g. on adherence to the five PD principles and the Accra Agenda for Action priorities; and the results of supplementary studies to fill gaps.

8. The Paris Declaration in Perspective Other international & national influences & forces Q1: PD in context Q1 Q2: Effects of PD on aid effectiveness Q3 Q2 Q3: Effects of PD on development results Q4 Conclusions: Compared against pre-PD or alternative approaches Overall development processes The Aid Partnership Aid influenced by PD commitments

9. The Core Questions (Refined through regional workshops and inputs from other IRG members) 1. “What are the important factors that have shaped and limited Paris Declaration implementation and its potential effects on aid effectiveness and development results?” (The Paris Declaration in context) 2. “To what extent and how has the implementation of the PD led to an improvement in the efficiency of aid delivery, the management and use of aid and better partnerships?” (Process and intermediate outcomes) 3. “Has the implementation of PD strengthened the contribution of aid to sustainable development results? How?” (Development outcomes)

10. Framework for conclusions • To what extent have the 5 principles of the PD been observed and implemented and the Accra Agenda priorities reflected? Why? Have there been conflicts or trade-offs between them? • What has the Paris Declaration achieved for aid effectiveness and development results? How significant are these contributions? How sustainable? Is there evidence of better ways to make aid more effective and contribute more to development results? • What has Paris Declaration-style development cooperation added compared with the pre-PD situation and alongside other drivers of development in the country, other sources of development finance and other development partners not so far endorsing the Declaration? • What are the key messages for a) national stakeholders, and b) donor countries and agencies? • What are the key implications for aid effectiveness in the future, taking account of new challenges and opportunities (e.g. climate change) and new actors and relationships?

11. Core components of the methodology • Questions, and the framework for conclusions, follow the criteria of relevance, efficiency, effectiveness and sustainability; • Include assessment of “upstream” or precursor steps to Paris in 2005; • Not expect clear-cut or one-dimensional causality, seek “plausible contributions”; • Searching analysis of (vital) context; • Sub-questions: descriptive, analytical, normative and evaluative.

12. Common specifications and suggestions for country evaluations On: • Types of evidence and, where applicable, indicators • Anticipated availability and (probable) reliability of data sources • Proposed sources, methods & techniques for data collection, analysis, triangulation and validation.

13. Anticipated methods (To be finalized at Inception Report stage) • Syntheses and meta-analyses of existing evidence; • The normal arsenal of evaluation tools and techniques (see p. 12); • Comparative studies, esp. on one “tracer” and other sectors; • Backward tracking to past Paris Declaration-like initiatives and their results; • Analysis of time-series data, and synthesis studies; • ‘Theory based’ (longitudinal) studies that are forward looking, incl. “direction of travel” and “distance travelled”; • Examine “mechanisms of change”: help explain results • Ensuring comparability (See p.12) • Ensuring validity and reliability (See p.13)

14. Accountabilities and Responsibilities • International Reference Group • National Evaluation Coordinator • National Reference/Advisory Group • Country Evaluation Team • Donor Headquarters Evaluation Team • Core Evaluation Team • Evaluation Management Group and Secretariat

15. Workplan and Schedule • See “Critical Milestones” Document

16. Proposed Core Evaluation Questions & Sub-questions Overview of sources of evidence for answering the Evaluation questions (elaborated in Generic TORs) Appendix A. The Overarching Evaluation Matrix for Phase 2