Norges Fredsråd seminar 16. januar 2008

1. Norges Fredsråd seminar 16. januar 2008 Resultatmåling av prosjekter Beate Bull Evalueringsavdelingen Norad

2. Hvordan resultatmåle fredsprosjekter eller sikre evaluability?Spørsmål som må stilles: • Hva menes med resultater i denne sammenheng? • Effekten av bistandsinnsatsen? • Inkluderes også uintenderte konsekvenser, negative eller positive? • Outputs / produktet av innsatsen? • Outcome nivå? • Impact (langtidsvirkninger)? • Hvilke resultater skal måles? Side/Page

3. What is evaluation? • Evaluation is used of assessments of an aid activity, program, policy, modality or strategy, carried out as systematically and objectively as possible. • The assessments can include the design of the activity, the program theory, its implementation and the results achieved. • Its purpose can be to determine the activity’s relevance, effectiveness, cost-effectiveness, impacts, and whether the intervention is sustainable. • Evaluations are primarily backward-looking (ex-post), but may also be performed for ongoing activities. • For more, see our Evaluation Guidelines athttp://www.norad.no/items/5704/38/7418198779/EvaluationPolicy2006-2010.pdf Side/Page

4. Premisser for måling av resultater: • En klar målformulering og klar formulering av forventede resultater • En klar beskrivelse av problemet en ønsker å påvirke/ og en konfliktanalyse • Beskrivelse av inputs, aktiviteter og outputs samt hvilke outcomes en ønsker å påvirke/bidra til • Identifisering av risiki • Identifisering av indikatorer og måter å verifisere disse på • Beskrivelse av eksterne og interne faktorer som en ikke har kontroll på • Beskrivelse av situasjonen ved oppstart / baseline Side/Page

5. Norads definisjon av resultater • outcomes i resultatkjeden; dvs endrete holdninger, atferdsmønstre eller livssituasjon, og • Impact dvs. de langsiktige / eller vidtrekkende samfunnsvirkningene Side/Page

6. Veileder for evaluering av konfliktforebygging og fredsbygging • Forsøker å ta innover seg utfordringene som eksisterer i forbindelse med å dokumentere hva som virker og ikke virker på fredsbyggingsfeltet. • Er aktuell for prosjekter og aktiviteter som jobber direkte med konfliktforebygging og fredsbygging Side/Page

7. Definisjon 'Conflict prevention and peacebuilding activities' are projects, programmes, policies, strategies or other interventions that adopt goals and objectives aimed at preventing conflict or building peace; they are usually (but not always) focused on a particular conflict zone – an area threatened by, in the midst of, or recovering from serious intergroup violence’. s. 16. Side/Page

8. Conflict prevention and peacebuilding work Side/Page

9. Summary: Key elements of planning an evaluation for those commissioning or preparing evaluations Key steps: • Define the purpose, objectives and use of the evaluation • Decide on the scope of the evaluation • Outline the key evaluation questions • Do or obtain a conflict analysis • Take timing and logistical issues into consideration • Co-ordinate with other actors • Consider conducting a joint evaluation • Select evaluation criteria • Devise evaluation management • Develop terms of reference • Select the evaluation team • Contracting Side/Page

10. Budget, Budget, Budget Side/Page

11. scope • Specifying the issues covered, • funds spent, • the time period, • types of interventions, • geographical coverage,,and • target groups • as well as other elements of the policy or intervention to be addressed. Side/Page

12. Scope continued: Delimitations • It is important to delimit the evaluation scope according to purpose, resources and time. • Also state which evaluation questions will not be sought answered. • Questions to be asked are: How far along the “results chain” (inputs activities  outputs outcomes  impacts) • Will the evaluation also cover inputs? • Will it look for immediate and long term impacts on institutions, on society or on broad conflict dynamics, and, if so, how? Side/Page

13. Select evaluation criteria • Relevance • Effectiveness • Efficiency • Sustainability • Impact Side/Page

14. Relevance • The peacebuilding relevance links the analysis of the conflict situation and the peacebuilding process with the intervention’s objectives and thus seeks to find out whether an intervention is on the right track to contribute to peacebuilding. Relevance of the intervention might change over time with changing circumstances. Side/Page

15. Evalaution Questions: Relevance • Is the intervention based on an accurate and up-to-date analysis of the conflict? Is it working on the right issues in this context at this time? • Does it, therefore, address relevant causes of conflicts, key dynamics and driving factors or key driving constituencies of the conflict? • Are the stated goals and objectives of the activity relevant to issues central to the conflict? Do activities and strategies fit objectives? • Has the intervention responded flexibly to changing circumstances over time? Has the conflict analysis been revisited or updated to guide action in changing circumstance. • What is the relevance of the intervention as perceived by beneficiaries and external observers Side/Page

16. Effectiveness • Effectiveness is used to evaluate whether an intervention has reached its intended objectives, with respect to its immediate peacebuilding environment in a timely fashion. The key to evaluating effectiveness and thus the linkage between outputs, outcomes and impacts, is finding out to what degree the envisaged objectives have been fulfilled and noting changes that the intervention has initiated or contributed to. Side/Page

17. Evaluation questions : Effectiveness • Has the intervention achieved its stated or implicit purpose, or can it reasonably be expected to do so on the basis of its outputs? • Is the effort achieving progress in a reasonable timeframe? Is it possible to speed the process? Should the effort be slowed down for any reason? • Does the intervention prompt people increasingly to resist violence and provocations to violence? • Do the stakeholders affected have a significant impact on the conflict? Are the right people / key people and/or many people being addressed? Were gender and relevant horizontal inequalities (ethnic, religious, geographical, etc.) taken into consideration? • Does the effort result in an increase in people’s security and in their sense of security? • Does the effort improve non-violent forms of conflict resolution or power management? • Does the effort result in meaningful improvement in relations among and within groups in conflict? • What major factors are contributing to achievement or non-achievement of objectives? Side/Page

18. Outcome evaluation • Outcome evaluation approach • This approach can be part of an evaluation or an entire assessment that focuses on outcomes of an intervention, i.e. the changes that had been achieved between outputs and impacts by the interventions, intended or unintended. The outcome evaluation referred to here works backwards from the outcome. They involve making judgments about the interrelationship between inputs and outputs on the one hand and outcomes on the other but do not start by analysing projects. This approach is suggested in the UNDP guidelines on outcome evaluation.[1] • [1] UNDP, 2002 Side/Page

19. Advantage of outcome evaluation • A perceived advantage with this approach for conflict prevention and peacebuilding evaluation is that it starts with what normally comes last in other evaluations: with analysing changes in the outcome. For instance: if the intended outcome was that journalists should change their style of reporting, and if it was found that there has been no change in reporting, then one could work backwards and find the reasons for this, and consider how to improve the chances of achieving this change in reporting. If the evaluated intervention has failed to contribute and other factors have been more important, this is an important lesson. Side/Page

20. Key questions for conflict and peace analysis • ProfileWhat is the political, economic, and socio-cultural context? What are emergent political, economic and social issues? What conflict prone/affected areas can be situated within the context? What are the geographic dimensions?Is there a history of conflict? • Conflict causes and potentials for peaceWhat are the structural causes of conflict? What issues can be considered as proximate or dynamic causes of conflict? What triggers could contribute to the outbreak/ further escalation of conflict?What strategies for dealing with conflict contribute to violence?What new factors contribute to prolonging conflict dynamics? What factors can contribute to peace? What factors are bringing peoples together? • Actors Who are the main actors? (people that perpetuate or mitigate the conflict)What are their interests, goals, positions, capacities and relationships? What capacities for peace can be identified? Who can make a difference?What actors can be identified as spoilers? Why? Are they inadvertent or intentional spoilers? • Dynamics and future trendsWhat are current conflict trends? Negative reinforcing cycles?What are windows of opportunity? What scenarios can be developed from the analysis of the conflict profile, causes and actors?How might different scenarios play out given likely future developments (short and long run)? Side/Page

21. Evaluation principles Evaluations need to be of high quality and shall be: • Objective, verifiable, and transparent: Evaluations shall be based on facts, which is to say reliable data or observations. • The results must be publicly accessible, not least to enable others to check facts and soundness of the analysis. • Impartial: Evaluations must provide a balanced view of strengths and weaknesses. In so far as different parties have conflicting views, the evaluation report should reflect and acknowledge these. • Independent: To ensure the greatest possible objectivity the Evaluation Department requires evaluations to be undertaken by independent professionals with no personal interest in the results; they must be thorough and their recommendations should be based on systematic collection and presentation of data. Side/Page