1 / 34

Dr. Sukai Prom-Jackson and Mr. George Bartsiotas Inspectors JIU January 29, 2016

THE ANALYSIS OF THE EVALUATION FUNCTION OF THE UNITED NATIONS SYSTEM Evaluation Practice in WIPO: Lessons and ways forward Internal Oversight Division (IOD) Evaluation Seminar. Dr. Sukai Prom-Jackson and Mr. George Bartsiotas Inspectors JIU January 29, 2016.

feliciac
Télécharger la présentation

Dr. Sukai Prom-Jackson and Mr. George Bartsiotas Inspectors JIU January 29, 2016

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THE ANALYSIS OF THE EVALUATION FUNCTION OF THE UNITED NATIONS SYSTEM Evaluation Practice in WIPO: Lessons and ways forwardInternal Oversight Division (IOD) Evaluation Seminar Dr. Sukai Prom-Jackson and Mr. George Bartsiotas Inspectors JIU January 29, 2016

  2. Introduction – Meeting of Minds for Dialogue A. Conceptual framework for the Evaluation function: • Definition of evaluation • Purpose of evaluation  • What defines the evaluation function? • The results framework and components of the function • The hardware: structures, systems, models, operations • The software: Institutionalization and culture - values, rules of the game, standards, capacities, leadership, incentives, risk management, double loop learning, delegation and engagement B. Theoretical framework for assessment of the function: • The systems framework: (i) Whole is more than sum of the parts; (ii) No system is stronger than its weakest link (Architecture/Governing bodies, management, staff/Evaluation Office and Evaluators) • Stages of development of the function: The JIU Maturity Matrix C. Findings: • What did we find? • What does it mean for your Organization? And what about your neighbors in Geneva?

  3. Definition: What is Evaluation? The judgment of value or worth of phenomenon or activities of the operations of United Nations System- What the UN System Values 1. Relevance 2. Effectiveness 5. Sustainability United Nations System values 3. Efficiency Coherence Partnerships 4. Impact

  4. The Purpose of EvaluationEvaluation supports: Accountability Learning and improvement Knowledge Development All of which are important for the continued relevance and value of the United Nations System

  5. The Evaluation Process and Methodology Systematic • Rigor, based on scientific principles: - Validity and reliability - Deductive, inductive, abductive logic • Grounded on management principles • Communication to enhance evidence – influenced decision making Related BUT DIFFERENT from: • Monitoring • Audit • Inspection/ Review • Knowledge Management

  6. About the Evaluation Function – Conceptual Framework for Assessment Demand of Evaluation Demand and Intentionality Organizational Context, Drivers, Level & Nature of Demand Supply, Adaptation and Growth Sustainability: Enabling Environment, Organizational & Institutional frameworks Relevance, Efficiency, Adaptability, Coverage and Readiness Credibility, Independence, Impartiality, Inclusion Quality, Validity and Reliability Results Utility, Use, Level of Use and Potential Impact

  7. Components of the Evaluation Function – Graphic illustration • Demand and Intentionality  • Context and drivers of demand • Nature and level of demand Results Supply, Adaptation, Growth The Enabling Environment Organizational & Institutional Framework for Evaluation Credibility: Independence, Impartiality, Inclusion, Transparency, Quality Relevance, Readiness Efficiency Adaptability Conditions in place to enhance Use Nature and level of Use Effect of Use-Impact Independence • Structural • Functional • Inclusion/ Transparency • Professional/ Technical • Behavioral Quality • Quality Reports • Quality Assurance • Competencies • Conditions to ensure quality • Coverage • Responsiveness • Readiness: -Decentralized Evaluation -UN Reform -National alignment -Gender and HR -Global challenges • Efficiency • Adaptability, improvement and growth •Recommendation implementation rates Accountability & Learning • Strategic decision-setting • Corporate use • Formative use and improvement • Development evaluation • Global/ National contribution • Report quality • Dissemination and communication strategy • Timeliness • Accessibility • Follow-up mechanisms • KM and Sharing • Role of Managers • Role of Governing bodies • Architecture • Governance • Institutional framework: -Mandates -Vision/Strategy -Policy -Norms/Standard • Guidance • Resources • RBM • Learning culture • Leadership • Organization change • Development results • UN goals(other) • Mutual accountability • Learning organization • Learning nations • Global governance • UN Leadership and Credibility

  8. The Architecture of the United Nations System 1. UN System wide Joint Inspection Unit 2. UN Subsystem-wide OIOS Inter-agency 3. Corporate Central/UN Organizations Part of policy, Strategy, Planning and Monitoring Part of Oversight-Separate Unit-Integrated Unit Stand Alone 4. Decentralized Evaluation/UN Organizations Embedded Departments and Secretariat Regional Offices Regional centers Technical Departments Country Offices level Programme Departments 5. Individual staff level: knowledge workers and communities/results driven/critical mind of inquiry

  9. The Evaluation Architecture of the UN System Systems Perspective (Overall System)

  10. Structural Arrangement of the Evaluation Function of the UN system

  11. Scope of the JIU Study • 28 United Nations system organizations • Organization profile: • Different mandates, sizes, structural arrangement • Examination of maturity level of the central function – 24 Organizations • Central Function Focus of study • Rapid assessment of decentralized – 10 Organizations • Three structures • Structured • Ad hoc/self evaluation • Bilateral donor model 12

  12. Maturity Levels of the Central Evaluation Function(Philosophical Foundation)

  13. General Findings and Challenges

  14. Findings and Challenges 5. Imbalances • Accountability versus learning and role of Executive Boards • Leadership also responsive /skewed to supporting accountability and limited focus on the organizational culture. • Role of central versus decentralized in context of changing world dynamics • All have evaluation policies – but absence of a strategy for the evaluation function in the context of organization – tendency for function to be removed = focus on responding to demand for accountability • Major weakness on use of evaluation (utility or instrumental use) and Factors affecting use • Second major weakness – readiness to support change and challenges – UNEG yet Silo mentality; internal focus not leverage opportunities; different mandates and effect on joint evaluation; generally low capacity

  15. Overalllevels of development and how WIPO is positionedof the central evaluation function for 28 JIU participating organizations Level 4 Transitioning to Level 4 UNDP Level 3 ILO (G) WFP UN Women UNIDO UNICEF UN OIOS UNFPA UNESCO Transitioning to Level 3 Level 5 FAO IAEAUNEP UNAIDS (G) UNODC WIPO(G) Level 2 ITC (G) WHO (G) UNCTAD (G) UN-HABITAT ICAO UNHCR (G) IMO WMO (G) UNRWA Level 1 ITU UNOPS UPU UNWTO

  16. Structural Location and Size in relation to Level of Development

  17. Ratings on key components of the evaluation function - WIPO and 24 UN system Organizations

  18. Performance of 28 participating organizations on the enabling environment Level 4 Transitioning to Level 4 UNDP ILO (G) Level 3 WFP UN Women UNIDO UNICEF UN OIOS UNFPA UNAIDS (G) UNESCO Transitioning to Level 3 FAO IAEAUNEP UNODC WIPO(G) Level 5 Level 2 ITC (G) WHO (G) UNCTAD (G) ICAO UNHCR (G) IMO WMO (G) UN-HABITAT Level 1 ITU UNOPS UPU UNWTO UNRWA

  19. The Enabling Environment and WIPO Progress Lagging behind Resources (6) Evaluation Architecture (4) Mandates, vision and policy (6.5) Governance (4.5) Supporting implementation of policy (6.3) Results and accountability/learning oriented (4.5)

  20. The Enabling Environment: WIPO and UN System Organizations

  21. Performance of 28 participating organizations on independence Level 4 Transitioning to Level 4 UNDP WFP UN OIOS UNFPA UN Women FAO IAEA ILO (G) UNIDO UNICEF WIPO (G) UNESCO Level 3 Transitioning to Level 3 UNCTAD (G) UNEP UNODC ICAOUNAIDS (G) WHO (G) Level 5 Level 2 UNHCR (G) ITC (G) IMO WMO (G) UN-HABITAT UNRWA Level 1 ITU UNOPS UPU UNWTO

  22. Findings on Independence and WIPO Progress Lagging behind Term of Head and rotation in the organization (8) Professional integrity of the function (5) Independence of budgetary process (3) Development and issuance of evaluation reports (8) Role of Member States (5)

  23. Elements of professional integrity and identity of the Evaluation function • Staff managing and conducting evaluation are evaluators (and also have expertise in in other discipline. Also have strategic thinking skills and are quick learners. • Evaluations address both performance (doing things right) and addresses critical evaluation questions of doing the right things, and strategic direction setting and positioning of the organization for added value and advancement. • The approaches and methods used follow professional methods for evaluation but there is judgment in application as well as innovation to enhance validity • The accountability for results is an integral part of role of function to enhance the learning culture ☐ There is independence but not isolation. There is engagement with the organization. ☐ Evaluation is not overshadowed by other disciplines or made compliant to other related disciplines (monitoring, research, audit, assessments etc.), thus not fulfilling its value added. ☐ Function has prominence or standing in the organization and with the governing bodies. ☐ The language of evaluation is fully recognized and organization uses this. ☐ When co-located, there is equivalent treatment with other functions in terms of resourcing, coverage, recognition, status and staffing. Also, there is value addition in the co-location ☐ Central Office or unit of Evaluation acts as custodian or steward or advocate for evaluation and engages organization on all aspects of the evaluation.

  24. Performance of 28 participating organizations on quality Level 4 Transitioning to Level 4 UNDP WFP ILO (G) Level 3 UNEP UN OIOS UNICEF UN Women FAO UNIDO Transitioning to Level 3 Level 5 UNODC ITC (G) UNFPA IAEA UNAIDS (G) UNESCO WIPO (G) Level 2 ICAO WHO (G) UN-HABITAT UNRWA UNCTAD (G) UNHCR (G) IMO WMO (G) Level 1 ITU UNOPS UPU UNWTO

  25. Findings on Quality and WIPO Progress Lagging behind Empirical/objective assessment (4) Quality of reports (5) Technical and managerial guidelines and tools (7) Consultant competencies (5) Controls and stakeholder involvement at various stages of the evaluation (7) Professional development of staff (5)

  26. Performance of 28 participating organizations on utility Level 4 Transitioning to Level 4 Level 3 UNDP UNESCO Transitioning to Level 3 FAO IAEAUNEP UNAIDS (G) UNODC ILO (G) WFP UN Women UNIDO UNICEF UN OIOS UNFPA Level 5 Level 2 ITC(G) IMO WIPO (G) WHO (G) UNCTAD (G) UN HABITAT WMO (G) UNHCR (G) Level 1 ITU UNOPS UPU UNWTO ICAO UNRWA

  27. Findings on Utility and WIPO progress Lagging behind Corporate/ summative use (4) Use for strategic decisions (4) Recommendation tracking system (8) Formative use (3) Use external to organization (3) Recommendation implementation rates (8) Effect of use on organizational effectiveness and value for the UN (1) Impact following implementation of recommendations (2)

  28. Readiness for Global changes and challenges Post 2015Ensuring Sustainability and Added Value

  29. Decentralized evaluation function models and archetypes in the United Nations system There are 3 models: • Ad hoc and Discretionary- Almost all United Nations organizations; • Well defined Institutional Framework Provided - Selected United Nations organizations: • UNDP: Regional Headquarters; Technical Bureaux; Regional Service Centre; Country Office • UNICEF: Same as UNDP but larger capacity for evaluation on the ground • UN Women: Regional advisors under the central Office (doing both); Country Office • Bilateral Donor Model

  30. The rising importance of the decentralized evaluation function of the United Nations system Evolution of the United Nations system Emerging need for the evaluation function in order to: • Rationalize • Restructure • Address complementarities • Deal with alignments With focus on: • Global context requiring transformations, restructuring and continuous evidence for fast action; • Sustainability and utility factors; Evolution of the function – Need for reform and integrated strategic direction setting of the evaluation functions

  31. Key findings for the decentralized evaluation function • The absence of an overarching and well-defined institutional framework makes the decentralized function tenuous. • Policies, norms and standards, and resources for decentralized evaluations are inadequate • Use of and learning from is limited by an organizational culture which is focused on accountability and responsiveness to donors

  32. Support from central function to the decentralized function - Strengths and Challenges for WIPO • Enabling conditions to support policy implementation • Evaluability and quality enhancement • Quality control, reporting, and compliance for corporate action

  33. Thank you!Questions? Preparation Team: Amanuel Zerihoun (Evaluation Officer) Gloria Villa (Intern) Yury Benyaminov (Intern)

More Related