ADB TA 4581 — Developing a National Result-Based M &E System for Key Projects 09-2006

ADB TA 4581 — Developing a National Result-Based M &E System for Key Projects 09-2006 Prof. Xue Yan, in behalf of TA 4581

TA 4581 Project Objective Strengthen capacity of Key Project Inspectorate Office which is under the Ministry of National Development and Reform Committee (KPIO) to develop result-based performance indicators of key national projects.

Background • KPIO created 1998: • major responsibility to monitor and inspect key national projects; • first inspections January 1999. • The magnitude of KPIO’s • >10,000 “national key projects”; • ~1,600 projects inspected up to 2005; • 85 staff (Beijing), ~500 (provincial & municipal KPIOs).

Background • KPIO’ s emphasis to date is on project implementation: • financial control • schedule achievement • quality of outputs • corruption detection • This TA will assist development of results-based indicators to supplement the current system, and raise knowledge and awareness of Results-based monitoring & Evaluation methods. A a more complete picture will be gained, to tell: • if the original purpose has been achieved; • how well the project was implemented; • how to improve operation of the finished project; • lessons for future projects.

Project Approach • Test results-based indicators through selected project case studies. • Train KPIO staff and government officials, using case studies. • Integrate selected indicators into KPIO system.

Results-based Monitoring and Evaluation Systems • Results The outputs, outcomes and impacts of a project. • Monitoring • Monitoring aims to ensure the project is on the right track and necessary adjustments are made to ensure successful outputs and outcomes. Monitoring also provides the basis for Evaluation, as it demonstrates project progress and points to possible adjustments to project plans and targets. • Monitoring is continuous from the beginning of a project. • Evaluation The systematic and objective assessment of an in-progress or completed project. Includes assessment of its design, implementation and results.

Results-based Monitoring Model Strategic goal Impact Need Outcome Inputs Outputs Activities

Strategic goal Strategic goal Impact Impact 战战略目略目标长长期效期效应应 Concept Operation Feasibility Need Need Outcome Outcome 需求需求直接直接结结果果 Design Bidding & Contracting Outputs Outputs Inputs Inputs Activities Activities 输输入入活活动动提交物提交物 The Model with the Project Cycle • In the past, project monitoring focused on implementation phase: activities and outputs. • Now governments increasingly focus on outcomes and impacts, to demonstrate value of projects in an accountable & transparent system. • Systems vary from country to country but all developed countries and many developing countries are working in the direction of Result-Based performance monitoring and evaluation.

Impact Statement of the long-term or strategic objectives. Outcomes Information on the expected direct results of the project being completed and put into operation. Outputs Physical deliverables of the project. Activities Major activities carried out to create the project deliverables. Inputs Project budget. The Framework The framework summarizes how the project will achieve results by converting inputs to outputs, and how outputs are expected to give the desired results.

R-B (addition) Nakproses (current) Results-Based Development Strategic goal Impact Need Outcome Inputs Activities Outputs

State Council Finance • NDRC • Ministry • Critically evaluate benefits • Prioritize future projects • Good governance Impact Outcome • Ministry • Department • KPIO • Project sponsor • Optimize performance • Minimize waste • Public accountability • Good governance • Performance culture • Benchmarking Outputs The interesting parties • Ministry • Department • KPIO • Project sponsor • Project manager • Contractor • Costs • Efficiency • Quality • Timeliness • Funding • Corruption • Public accountability

Benefits of Results-Based Monitoring • Optimize project outcome performance • Avoid mistakes from the past • Priorities future project proposals • Greater public accountability • Can demonstrate good governance • Have a performance culture • Minimized project waste • Set higher standards using Benchmarks

Results Monitoring Model Develop industry Industry grows Good transport access Road usage, Speedy, Smooth New road Materials Equipment Manpower Construct roads

Results Monitoring Indicators Industry grows Road is used, Fast, Smooth New road

What we want to know after construction ends and asset is being used • is the finished project being used as intended? • is the operating performance as planned or do we need to make some modifications to make it work better? • is the project achieving the intended results?

What we want to know later (>3 years after construction): • did the project achieve the intended operating results? • did the project achieve the intended impact? • should we do similar projects in future? • what lessons have we learned from this project that we can apply to future projects?

Outcomes • Usage rate • Average number of vehicles per day travelling on road. • Output of productive unit: • e.g. MWh provided to industry • Quality • Average travel time between A and B by road • quality of electricity output, e.g. no load shedding, xx MW peak load maintained • Financial • revenue from the asset usage (calculated as pay-back period or rate of return FIRR)

Outcome indicator: Usage

Outcome indicator: Usage What does this tell us about the project?

Outcome indicator - Usage What does this tell us about the project?

Outcome indicator: Operating Revenue

Outcome indicator: Operating Revenue What does this tell us about the project?

Outcome indicator: Operating profit

Outcome indicator: Operating profit What does this tell us about the project?

Case Studies – Objectives • Objectives: • Pilot Result-Based performance indicators, to bring international best practices into project monitoring and evaluation • Find practical ways of implementing them into the current computer system • Develop cases and materials for promotion of and training in Result-Based Monitoring & Evaluation • Provide learning experience with Result-Based performance indicators for KPIO, KPIO provincial level and executing agencies

Considerations for case study projects • Sectors • Power • Transport • Hospital • Others • Status • Commenced project implementation • Recently completed • Completed 3 – 5 years • Qualities • Good or OK • Problematic

Desirable variety of case study projects • Contract types • turn-key • Build-Own-Transfer • multiple contractors/consortium • etc… • investment sources • Government • International loans • Private or foreign investment • other • large/small • profit/non-profit

Steps • Choose case study projects. • Gather documentation on project justification • Develop monitoring framework for each project. • Define results-based indicators for each project. • Collect data for result-based indicators for each project. • Assess indicators, change if necessary. • Use case studies to develop training materials.

Case Studies • The project indicator monitoring framework • The project indicator working sheet • The main activities • The results (Hebei Expressway) • Some issues and findings in the case studies.

Thank you! 谢谢！

ADB TA 4581 — Developing a National Result-Based M &E System for Key Projects 09-2006