1 / 26

IDB ’ s Evaluation Architecture for Public and Private Sector Project Evaluation

IDB ’ s Evaluation Architecture for Public and Private Sector Project Evaluation. Cheryl Gray, Director Office of Evaluation and Oversight (OVE) Inter-American Development Bank. Presentation at the meeting of Sub-Regional Development Institutions Johannesburg, 28 April 2014.

mooreryan
Télécharger la présentation

IDB ’ s Evaluation Architecture for Public and Private Sector Project Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IDB’s Evaluation Architecture for Public and Private SectorProject Evaluation Cheryl Gray, Director Office of Evaluation andOversight (OVE) Inter-AmericanDevelopmentBank Presentation at the meeting of Sub-Regional Development Institutions Johannesburg, 28 April 2014

  2. Some background on IDB • The IDB Group includes IDB, the Inter-American Investment Corporation (IIC), and the Multilateral Investment Fund (MIF) • 26 client countries and about $12b in lending per year • Biggest borrowers: Argentina, Brazil, Colombia and Mexico • Over 80% lending is to public sector (sovereign-guaranteed) • About $150m in technical cooperation per year • funded by ordinary capital and trust funds • About 2000 employees and $600m administrative budget

  3. IDB’s Accountability System Board of Executive Directors Independent Consultation and Inspection Mechanism Office of Evaluation and Oversight IDB Management Management’s Internal Accountability Mechanisms • Self Evaluation • Management oversight • Institutional Integrity • Internal Audit • Ombudsperson

  4. OVE is independent from IDB Management • REPORTING: OVE Director selected by Board for 5-year term (renewable once); reports directly to the Board; can be removed only by Board and only on performance or ethical grounds • BUDGET: “below-the-line” and approved by Board; currently 28 staff, 15 Research Fellows • EVALUATION PROCESS: formal protocol for management and Board review • ACCESS TO INFORMATION: OVE has full access to all IDB information • RECOMMENDATION FOLLOW-UP: Online system tracks adoption of OVE recommendations • CONFLICTS-OF-INTEREST: Evaluators follow conflict-of-interest rules • OUTREACH: OVE handles its own external outreach • IDB POLICIES: OVE follows all IDB financial management and HR policies and procedures

  5. OVE seeks to address both accountability and learning • Accountability for results: • Is the Bank doing what it says? Is it achieving its objectives? To what extent is it contributing to country outcomes? With what results and development impact? • Learning: • What projects and programs achieve results – relevance, efficacy, and efficiency – and why? • How can M&E be strengthened in the Bank and in client countries?

  6. Notes, summaries Thematic, sectoral, corporate evaluations Country program evaluations Project-level evaluations OVE produces a range of evaluation products

  7. Several tools are used for project evaluation in IDB • Development Effectiveness Matrix (DEM): evaluability ex-ante • Project Monitoring/Supervision Reports (PMR/PSR): monitoring during implementation • Project Completion Reports (PCR) [public sector] or Extended Project Supervision Reports (XPSR) [private sector]: self-evaluation (ex-post) • Independent evaluation (OVE) • DEM, PCR, XPSR validations • Comparative project evaluations

  8. To date public and private projects have been evaluated using different criteria in MDBs

  9. …and using somewhat different approaches and standards • Basis of the evaluation • Public sector: objectives-based • Private sector: commercial benchmarks • Attribution and additionality • Environmental and social safeguards • Definitions of rating scales and what is required for a satisfactory rating

  10. IDB is moving toward more unified criteria and guidelines • Common basis of evaluation -- the project’s “objectives” (i.e. intended results) • Common criteria that are conceptually consistent, with some sub-criteria specific to public and private sector operations • Common guidance on evidentiary requirements, analytical methods, and benchmarks for a positive rating • Common approaches to attribution The goals are comparability and a centralfocus on development results in all projects.

  11. Other ongoing challenges to address • The quality of PCRs has been uneven but is now improving (pilot program) • The project monitoring system changed recently and may need further adjustments • The evaluation framework for nonlending work is less-developed than for projects (in IDB and other MDBs) • Our goal is to continue to strengthen IDB’s focus on results.

  12. Other ongoing challenges to address • The quality of PCRs has been uneven but is now improving (pilot program). • The project monitoring system changed recently and may need further adjustments. • The evaluation framework for non-lending work is less-developed than for projects (in IDB and other MDBs). Our goal is to continue to strengthen IDB’s focus on results.

  13. Thank you www.iadb.org/evaluation

  14. Extra slides

  15. Relevance • Consistency of the project’s intended results with beneficiary needs, country priorities, and IFI assistance strategy and corporate goals • Justification for public sector involvement • Evidence-based justification for market failures and targeting • Logic linking project activities to intended intermediate and final outcomes

  16. Public Sector Private Sector Achievement of Outputs Fulfillment of Project Business Objectives Achievement of Outputs • The extent to which the project achieved its targeted outputs (appraisal projections or performance targets) • The extent to which the project delivered on its process and business objectives (private sector) • Attribution analysis not needed

  17. The project’s contribution (or expected contribution) to its intended development, policy, or transition outcomes • The project’s reach to intended beneficiaries • The project’s success in reducing or compensating for market failures • The project’s impact on all stakeholders (employees, suppliers, competitors, etc.) (private sector)

  18. Outcomes are assessed against a without-project counterfactual, either using an impact evaluation or (more commonly) a theory-based approach to establish plausible causality • The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether observed outcomes moved in the “right” or “wrong” direction • On an exceptional basis, “Not Rated” is possible when evidence is missing or weak

  19. Rating Based on Results Attributableto the Project Observed Observed decrease in outcomes indicator What would have happened without the project (counterfactual) Outcome attributable to the project Counterfactual Project Start Project End

  20. The project’s contribution to broad corporate goals that were not included in the project’s intended results (e.g., rural poverty reduction, shared prosperity, gender equality) • Uses a theory-based approach to establish plausible causality • The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether observed outcomes moved in the “right” or “wrong” direction • “Not Rated” is a possible rating

  21. Positive or negative results of the project that were not reflected in its intended outcomes (e.g., environmental damage) • Uses a theory-based approach to establish plausible causality • Unintended outcomes must be at least as well evidenced as the project’s intended outcomes • “Not Rated” is a possible rating

  22. Public Sector Private Sector Efficient Use of Resources Financial Performance Economic Efficiency Implementation Efficiency Economic Efficiency Implementation Efficiency • The profitability of the project from the perspective of its financial stakeholders (privatesector) and society at large (public and private sector) • The financial performance of the project/company is compared to a without-project counterfactual (private sector) • The economic efficiency of the project (CBA or CEA) is compared to a without-project counterfactual (public and private sector) • Also considers implementation efficiency (e.g., delays)

  23. Public Sector Private Sector Sustainability Outcome Sustainability Commercial Sustainability Compliance with Safeguards Outcome Sustainability Compliance with Safeguards • The sustainability of outcomes achieved or expected to be achieved (public and private sector) • The commercial sustainability of the company, financial institution, and/or sub-borrowers/fund investees (mainly private sector) • The environmental and social sustainability of project results, measured by compliance with relevant safeguards (public and private sector)

  24. Relevance Efficient Use of Resources Sustainability Overall Project Performance Rating

  25. Public Sector Private Sector IFI Performance Quality at Entry Quality of Supervision Quality at Entry Quality of Supervision Client Performance Non-Financial Performance of the Company Gov’t. and Implementing Agency Performance IFI Investment Profitability

More Related