Independent Evaluation « Quality Assessment of SDC’s External Evaluation Reports »

1. Independent Evaluation « Quality Assessment of SDC’s External Evaluation Reports » Anne BichselEvaluation + Controlling Division

2. Content • Objectives of the Quality Assessment • Procedures and Method • Findings • Recommendations

3. 1. Objectives ot the Quality Assessment • To assess the quality of SDC‘sexternal evaluation reports • To contribute to improving future performance and achieving a higher evaluation quality of SDC‘s external evaluations

4. 2. Procedures and Method (I) • Step I: Random Sample of 12 external evaluation reports • Step 2: devise a list of evaluation standards on the basis of- „DAC Minimum Sufficient Evaluation Standards“- SEVAL Standards- Key questions in Approach Paper

5. 2. Procedures and Method (II) • List of evaluation standards with 4 categories • Utility (9 standards): readable, accessible, timely evaluations with a good summary • Feasibility (3 standards): realistic well thought-out evaluations • Propriety (4 standards): ethical aspect of evaluations • Accuracy (7 standards): proper scientific methods and procedures

6. 2. Procedures and Method (III) • Step 3: review documents, conduct interviews with desk managers and evaluators • Step 4: work through all the standards for the sample of evaluations. Result: 12 fact sheets • Step 5: analysis and comparison • Step 6: conclusions and recommendations

7. 3. Findings (I) • Strengths of external evaluation reports • Identification and participation of stakeholders • Timely reporting • Cost effectiveness • Complete and balanced assessment

8. 3. Findings (II) • Weaknesses of external evaluations reports • Selection procedures of evaluation team • Formal written agreement • Comprehensive and clear reporting • Description of evaluation purpose and objectives • Description of evaluation procedures and methods • Making findings available

9. 4. Recommendations (I)  Draft good, realistic and comprehensive TORs • Clear purpose • Clear, focused, concise and understandable objectives with three to four questions • Agreement on TORs among SDC divisions • Realistic expectations in terms of resources and timeframe • Beware of questions at the impact level

10. 4. Recommendations (II)  Ensure more competitive and open selection procedures • Aim at balance in desired knowhow and independence of evaluators • Place more weight on evaluation knowhow (methods) • More open and transparent selection procedures • More competition • Careful team building in matching personalities, skills, division of labor

11. 4. Recommendations (III)  Improve the conditions for the utilisation of external evaluations • Accompany evaluations closely for maximum learning potential • Continuity on the part of desk managers • Reports need to be more accessible to a wider audience • Enhance the quality of evaluation reports • Make results of external evaluations more widely available

12. 4. Recommendations (IV)  Enhance the conditions for high quality external evaluations Evaluations are a challenging business, hence:Provide support and training for desk managers • Training in evaluation • Support in drafting TORs and commissioning evaluations • Support in obtaining high quality reports

13. In sum: less is more !

14. Direktor DEZA Stellv. Direktor Gerhard Siegfried Samuel Wälty Anne Bichsel Sekretariat (je 50%) Regula Herlan Christa Rohner jeweilige Bereichsleitung H-Bereich O-Bereich E-Bereich M-Bereich F-Bereich A-Bereich Christoph Jakob Kuno Schläfli Peter Meier Regula Bäbler MarkusGlatz Ivo Angehrn Alexandre Kobel Follow-up by the Core Learning Group: Evaluation & Controlling Net

15. TORs&contracts • Implementation • Draft Report • Debriefing • Final Report 0. • COSTRA Decision Steps of an Independent EvaluationCOSTRAComité Stratégique I. • Approach Paper • Core Learning Partners (CLP) II. III. • Agreement at Completion Point • Dissemination to Broad Learning Partners (BLP) IV. • Senior Management Standpoint discussed in COSTRA

16. Measures to improve quality • Communicated evaluation results / recommendations in all departments • Clarified and communicated the palette of instruments • Developing toolkit for commissioners of evaluations • Intraweb: Description of Process / examples • Principles to ensure successful Use of Evaluations • Improving training courses: curriculum and availability (1 day training Ext. Evals) • Backstopping and Quality Control by the E&C Net

17. Palette of Instruments Independent Evaluation External Evaluation External Review Self-Evaluation Expert Opinion

18. ExternalEvaluations Independent Evaluationssince 2001 ExternalReviews according to operational needs 5 – 10 annually 2 – 5 annually outside line management( E&C Division) within line management - issues of interest across departments- selected country programs - to generate knowledge within the department- of interest across divisions - ongoing operations - in the context of program cycle management aim for the same standards DAC Executive Summary Registration in data management system (DMS)