Evaluering – oppskrift på TOR

1. Bistandstorget Evaluering – oppskrift på TOR w ww. R e s u l t a t s t y r i n g . n o Anette P. Simonsen 24. oktober 2012 9:00-16:00 anette@resultatstyring.no

2. INTRO Evaluering som en del av M&E w ww. R e s u l t a t s t y r i n g . n o

3. Monitoring and Evaluation Mål- og resultatstyring Anette P. Simonsen

4. Definitions M&E Monitoring: • refers to the regular follow-up and assessments during the implementation of a project or programme. • normally provides information on expenditure, activities and outputs and a certain extent outcome • is used for adjusting the course of the project/programme as well as writing progress reports. Evaluation • is the systematic and independent assessment of a project/ program that is performed at certain stages of the project (e.g. mid-term, end of project). • would provide information on output,outcomeand impact • is used for adjusting the course or assessing continuation or ending a project. Mål- og resultatstyring Anette P. Simonsen

5. M&E definitionagain.. Evaluationapproaches often have high methodological standards, a separate budget and often scientific requirements for measurements and analysis. Monitoringon the other hand is a routine task, regularly ‘checking up on how things are’ and recording this information in a way that is useful both w ww. R e s u l t a t s t y r i n g . n o ObservingChange N-P-Aid

6. Norad Regelverkfor støttetilsivilsamfunnsaktører http://www.norad.no/no/tilskudd/s%C3%B8k-st%C3%B8tte/_attachment/153487?=true&_ts=12fd9e2bf57 Se spesielt side 3: «Beslutning om støtte skal tas etter en helhetsbedømmelse av søknaden. Hovedsakelig er kriteriene delt i to; de som gjelder for vurdering av organisasjonen og de som gjelder vurdering av bistandsvirksomheten. Sentrale punkter i vurderingen av organisasjon er: • Kapasitet og kompetanse til å nå organisasjonens mål • Organisasjonens kostnadseffektivitet, det vil si evnen til å gjennomføre organisasjonens mål med lavest mulige kostnader. • Systemer for resultatoppfølging • Evne til å identifisere og håndtere risiko w ww. R e s u l t a t s t y r i n g . n o

7. Økonomireglementetsier: § 16 Evalueringer • Alle virksomheter skal sørge for at det gjennomføres evalueringer for å få informasjon om effektivitet, måloppnåelse og resultater innenfor hele eller deler av virksomhetens ansvarsområde og aktiviteter. Evalueringene skal belyse hensiktsmessighet av eksempelvis eierskap, organisering og virkemidler, herunder tilskuddsordninger. • Frekvens og omfang av evalueringene skal bestemmes ut fra virksomhetens egenart, risiko og vesentlighet. w ww. R e s u l t a t s t y r i n g . n o

8. M&E har 3 formål M&E skal bidra til: • Kontroll av virksomheten - Forsikre interessenter om at midlene blir brukt på riktig måte, for avtalte mål og at de ulike samarbeidspartene oppfyller sine forpliktelser • til beslutningsprosesser – gi informasjon som underbygger de valg organisasjonene står overfor, når de må fatte beslutninger om prosjekter/ programmer skal fortsette, avsluttes eller endres • til å fremme organisatorisk læring – ved å gi aktuelle aktører kunnskap og informasjon om prosjekter/ programmer samt den sammenhengen de virker inn i. Ingen begrensning i hva som er mulig læring. w ww. R e s u l t a t s t y r i n g . n o Kruse & Forss (2000)

9. Preparing for Evaluation Look back at: • Does the project/programme have a clear goal hierarchy and a set of M&E plans? • Have the plans been monitored? • Are there good and regular monitoring reports? • Are the project staff available /still contactable? • Are the stakeholders involved? • Are the donor(s) involved? w ww. R e s u l t a t s t y r i n g . n o From Local Livelyhoods

10. Action Plan for Evaluation - Approvingtheactivity and thebudget w ww. R e s u l t a t s t y r i n g . n o

11. Action Plan versus TOR The Action plan • Formulates the internal “project” of performing an evaluation • It documents the approval of the decision to perform an evaluation and the budget for it. The TOR • is a work description for the team performing the evaluation • It constitutes only one part of the Action Plan w ww. R e s u l t a t s t y r i n g . n o

12. Action Plan for Evaluation • Background and justification • Need for evaluation at this point • Previous relevant evaluations / reviews • Who will be involved in evaluation / cooperation with other stakeholders/donors on the evaluation • Target group (audience) • Objective of evaluation • Any special requirements to team composition, method etc.? • Outputs/main activities • Planning phase • Implementation • Follow up • Risk and uncertainties • Organisation and responsibilities • Budget w ww. R e s u l t a t s t y r i n g . n o

13. Donor Justification for evaluation Donor Donor Donor Norwegian NGO Partner Tanzania Stake-holders Stake-holder Project w ww. R e s u l t a t s t y r i n g . n o IMPLEMENTATION LEVEL Stake-holders Stake-holder

14. 1. Background and Justification Do we really need an evaluation at this point? What is the problem we are trying to solve? E.g. • The M&E system is weak so we need to increase level of information on progress and or achievement of results? • The target group is complaining on quality of services? • We need to secure funding by a more objective voice on achievement of results? w ww. R e s u l t a t s t y r i n g . n o

15. M&E in project Cycle w ww. R e s u l t a t s t y r i n g . n o

16. When? Reporting to donors Example from 5 yearprogramme: w ww. R e s u l t a t s t y r i n g . n o

17. Why? Status report Norad • 3.1 Describe the reviews/evaluations that have been carried out during the year • In what way(s) did the review/evaluation contribute to learning and improvement of the programmes? • Are external reviews submitted to Norad’s data base for evaluations?(state type of evaluation – external, internal or mix external/internal) w ww. R e s u l t a t s t y r i n g . n o

18. Timing of evaluations Appraisal/ evaluation – before start-up Mid-Term Evaluation – towards middle of programme period End-Term evaluation – at the end of programme Formative evaluations – during programme Summative evaluations – at the end Ex-post evaluations – some time after Impact evaluations – at the end – some time after w ww. R e s u l t a t s t y r i n g . n o

19. OECD evaluation on DAC principles 1998: • Evaluations most useful when they are linked to current and future programmes rather than historical perspectives (although accountability requires the latter type) w ww. R e s u l t a t s t y r i n g . n o

20. 2. Who will be involved in evaluation / coop with other stakeholders Discuss evaluation plans and design with the stakeholders (staff, partners, benficiaries, others) and include the primary target group. • Who in the commissioning organisation is familiar with project/programme • Who are the stakeholders? • Which other donors are involved in Project/programme or even thematic area og geographical area. • Others? w ww. R e s u l t a t s t y r i n g . n o

21. 3. The target group (audience) Who will read, learn from and use the evaluation report should be one of the a determining questions in developing the terms of reference and designing the evaluation. In other words who is the primary target group that should be able to make use of the report? Should e.g. Norwegian NGO / the partner /stakeholders / members / taxpayers/ donors make use of the evaluation? w ww. R e s u l t a t s t y r i n g . n o

22. OECD evaluation on DAC principles 1998: • Evaluations are performed mostly because donors ask for it. Little attention is paid to the demand, the use and usability of evaluations, and differentiation according to different audiences w ww. R e s u l t a t s t y r i n g . n o

23. 4. Objective of the evaluation The purpose of evaluations is normally a combination of a need for control (accountability) to contribute to decision making and for learning • Partner – all 3 • Norwegian all 3 • Donor…. Who cares? w ww. R e s u l t a t s t y r i n g . n o

24. Objective of Evaluation • The implementing partners will have increased knowledge and lessons learned in their continuous work within the thematic areas in their respective countries. • The Norwegian NGO will be better equipped for future disasters of similar order of magnitude, both in the initial and recovery phase. • The private and corporate donors and the Ministry of Foreign Affairs will gain knowledge about the results of their donations. w ww. R e s u l t a t s t y r i n g . n o Who is the target grouphere?

25. Objective of Base line study The main objectives of the baseline survey are: • To get an initial baseline standing/situation/ context of the current and existing governance and livelihood situation, systems and structures in the impact areas • To establish baseline indicators for the programme that will imply a clear monitoring and evaluation plan (M&E) and platform for impact evaluation (IE) • To source information that is required to describe qualitatively and quantitatively the outcome indicators and output indicators for the Caritas Malawi programme on governance and livelihood. w ww. R e s u l t a t s t y r i n g . n o Who is the target grouphere?

26. Objective of Evaluation • Have conversations on the internal functions, including financial management, in each organisation involved in the visit in order to strengthen the organisation • Have conversations about the cooperation (communication, interaction) between the organisations in order to improve cooperation • The review should be summarized in a report in which the strengths and weaknesses are described, and proposed changes in procedures and communication within and between organisations are described. w ww. R e s u l t a t s t y r i n g . n o Who is the target grouphere?

27. Objective of Evaluation • Conducting a mid-term external review in 2011 will establish an overall picture of whether the five-year programme is on track towards 2013 and create a basis for corrective actions and the planning ahead for the final 2 years of implementation. The review will be an input to a mid-term meeting we want to hold in the second part of 2011. w ww. R e s u l t a t s t y r i n g . n o Who is the target grouphere?

28. Objective of Evaluation Formålet med evalueringen er å: • å gi LNU og bevilgende myndigheter økt kunnskap om i hvilken grad de tiltakene som får støtte fra LNU bidrar til å nå støtteordningens prosjektmål (måloppnåelse). • å gi LNU og bevilgende myndigheter økt kunnskap om eventuelle ikke-planlagte effekter (positive eller negative) av de tiltakene som får tilskudd. • å gi LNU et grunnlag for å videreutvikle sin forvaltning av støtteordningen. Formålet med evalueringen er avgrenset på bakgrunn av de økonomiske rammene for evalueringen. w ww. R e s u l t a t s t y r i n g . n o Who is the target grouphere?

29. Organisational review: w ww. R e s u l t a t s t y r i n g . n o PREBEN LINDØE for BN 2006

30. 5. Any special requirements to Team, method etc. Team: Internal / external Special expertise necessary Method: Level of data collection Special tools to be used NB! The action plan should not be too detailed at this point. Difficult to change. w ww. R e s u l t a t s t y r i n g . n o

31. 6. Outputs/ main activities The Action plan is split into • Planning phase • Implementation • Follow up w ww. R e s u l t a t s t y r i n g . n o

32. Exampleof Outputs and activities in Action Plan Didyouforgetabout Learning?

33. 7. Risk and uncertainties Any project meets uncertainties: e.g w ww. R e s u l t a t s t y r i n g . n o

34. 9. Organisation and responsibillities w ww. R e s u l t a t s t y r i n g . n o

35. Læring – referansegruppeevalueringsavdelingennorad Det er evalueringsavdelingen sin målsetning at evalueringsprosesser skal være deltakende og inklusive og fremme størst mulig læring. Det etableres derfor ofte en referansegruppe bestående av interessenter, berørte parter, ev. faglige ressurspersoner, fagavdeling og evalueringsavdelingen. Disse kommenterer arbeidet underveis. I den grad det er hensiktsmessig ønsker også evalueringsavdelingen å arrangere seminarer underveis og i etterkant. w ww. R e s u l t a t s t y r i n g . n o

36. OECD evaluation on DAC principles • There are reason for concern regarding the way in which lessons and experience, gained during the evaluation process, are transmitted to current operation managers. w ww. R e s u l t a t s t y r i n g . n o

37. 10. Budget example 1 w ww. R e s u l t a t s t y r i n g . n o

38. 10. Budget example 2 w ww. R e s u l t a t s t y r i n g . n o

39. TOR – Terms ofreference w ww. R e s u l t a t s t y r i n g . n o

40. Action Plan versus TOR The Action plan • Formulates the internal “project” of performing an evaluation • It documents the approval of the decision to perform an evaluation and the budget for it. The TOR • is a work description for the team performing the evaluation • It constitutes only one part of the Action Plan w ww. R e s u l t a t s t y r i n g . n o

41. Tor Norad 1 Background • Short description of the programme that will be reviewed, based on the Agreement, Programme Document(s) and appraisal • Why the review is initiated • Team composition and leadership. (e.g. Embassy/Norad team, joint donor/Partner, external team or combination of these) 2 Purpose, context and intended use • Description of the main purpose, context and intended use (stakeholders) 3 Scope of work Type of assessments may include: (for definition of the review criteria, see Annex III) • Efficiency • Effectiveness • Impact (if the programme has been operating for some years) • Relevance • Sustainability • Risk management • Particular concerns to be investigated • Audit • Anti-corruption measures 4 Implementation of the review • Sources of information and methodology to be employed • Division of responsibility between the consultant/team, the Embassy, other donors and the Partner(s) • Timetable for preparation, field work and finalisation of report • Budget 5 Reporting • Description of required report format • The need for an introduction summary with main conclusion on lessons learned and recommendation(s) • Report in electronic form and/or paper, language w ww. R e s u l t a t s t y r i n g . n o

42. Evalueringssprinsipper (EvalNorad web) Evalueringsprinsipper • Objektive, etterprøvbare og gjennomsiktige: Vurderinger skal være basert på faktiske utsagn; utsagn skal være basert på troverdige (reliable) data eller observasjoner. Relevante interessenter i Norge eller mottakerlandet skal konsulteres i forbindelse med evalueringers tilrettelegging og gjennomføring, inklusive i utarbeiding av oppgavebeskrivelse og diskusjon av rapportutkast. • Upartiske: Evalueringer skal gi en avbalansert fremstilling av styrker og svakheter. I den grad interessenter har forskjellige synspunkter skal disse tas med i evalueringen. • Uavhengige:  Medlemmer av evalueringsteamet må ikke ha vært personlig engasjert i de aktiviteter som skal evalueres. Virksomheter som gjennomfører evalueringer må ikke ha vært involvert i forberedelse eller gjennomføring av aktivitetene. w ww. R e s u l t a t s t y r i n g . n o

43. Description of Project /issue to be evaluated w ww. R e s u l t a t s t y r i n g . n o

44. Description of project to be evaluated Basically a summary of project document • Especially important is the goal hierarchy and any changes in that along the way • M&E System must be described and commented on • Previous or other relevant evaluations should be mentioned here • Budgets must be included too. w ww. R e s u l t a t s t y r i n g . n o

45. Scope of evaluation • The boundaries of the evaluation (e.g time period, area) • Evaluation criteria and evaluation questions w ww. R e s u l t a t s t y r i n g . n o

46. Scope of evaluation The formulation of “Evaluation questions” is one method that will end up with a precise definition of what should be evaluated. Evaluation questions represent “what one wants to know through evaluation.” • When formulation the evaluation questions it is useful to consider which of the DAC Principles for Evaluation of Development Assistance that should be covered w ww. R e s u l t a t s t y r i n g . n o

47. One goal -LFA spreadout Development Goal (expected Impact) Project Goal (expected Outcome) OUTPUT 1 OUTPUT 2 OUTPUT 3 OUTPUT 4 Mål- og resultatstyring Anette P. Simonsen Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Inputs Inputs Inputs Inputs

48. DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) • efficiency, • effectiveness, • impact • relevance, • sustainability. w ww. R e s u l t a t s t y r i n g . n o

49. Evalueringskriterier(web Norads evalavd) • Den enkelte evaluering bør normalt belyse følgende: • Relevans: Er aktiviteten relevant i forhold til norsk bistandspolitikks målsetninger og strategier? Er aktiviteten relevant i forhold til mottagerlandets behov og prioriteringer? Er den relevant i forhold til det utviklingsproblem den skal se på? • Måloppnåelse (effectiveness): Er de primære målsetninger for aktiviteten nådd? Er de planlagte resultater nådd? • Bærekraft (sustainability): Hva er langtidsvirkningen av aktiviteten? Vil aktiviteten kunne videreføres også etter at bistandsfinansieringen er avsluttet? Er det lokalt eierskap? • Produktivitet (efficiency): Er investeringene og driftsomkostningene berettighet? Kunne de samme resultater ha blitt oppnådd med færre midler? • Virkning (impact): Positive og negative primære og sekundære langtidseffekter produsert av aktivitetene, direkte eller indirekte, tilsiktede eller utilsiktede. w ww. R e s u l t a t s t y r i n g . n o

50. DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Evaluation criteria: Efficiency Effectiveness Impact Relevance Sustainability w ww. R e s u l t a t s t y r i n g . n o