Methods for Assessing Policy Impact

1. Methods for Assessing Policy Impact Process and Partnerships for Pro-Poor Policy Change, Project Initiation Workshop 1 ILRI, 21st February 2005

2. Workshop Outline • Introduction to the RAPID Framework and ILRI/ODI Project • Case Study Approach • Episode Study Approach • Outcome Mapping Approach • RAPID Outcome Assessment (ROA) Approach Lunch • Practical Sessions

3. Workshop Purpose & Objectives Objectives By the end of the workshop, participants will: • understand the Process and Partnership for Pro-Poor Policy Change project’s purpose and general approach • have the opportunity to contribute their own suggestions to improve the project; • understand, and have had the chance to try out the three key methods which will be used in the project; • assess the usefulness of the approaches in their own work. Purpose: To familiarise the participants with the general approach and specific methods to be used in the SDP case study

4. An introduction to the RAPID Framework and ILRI/ODI Project

5. Definitions • Research: “any systematic effort to increase the stock of knowledge” • Policy:a “purposive course of action followed by an actor or set of actors” • Agendas / policy horizons • Official statements documents • Patterns of spending • Implementation processes • Activities on the ground

6. Policy Processes - Identify a policy problem - Commission research - Assess the results - Select the best policy - Establish the policy framework - Implement the policy - The problem is solved

7. Reality… • “The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies1” • “Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2” 1 - Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in Agricultural and Rural Development, Heineman Educational Books, London 2 – Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges, International Service for National Agricultural Research (ISNAR) Research Report No 21

8. Linear model Percolation model, Weiss Tipping point model, Gladwell ‘Context, evidence, links’ framework, ODI Policy narratives, Roe Systems model (NSI) External forces, Lindquist ‘Room for manoeuvre’, Clay & Schaffer ‘Street level bureaucrats’, Lipsky Policy as social experiments, Rondinelli Policy Streams & Windows, Kingdon Disjointed incrementalism, Lindquist The ‘tipping point’, Gladwell Crisis model, Kuhn ‘Framework of possible thought’, Chomsky Variables for Credibility, Beach The source is as important as content, Gladwell Linear model of communication, Shannon Interactive model, Simple and surprising stories, Communication Theory Provide solutions, Marketing Theory I Find the right packaging, Marketing II Elicit a response, Kottler Translation of technology, Volkow Epistemic communities Policy communities Advocacy coalitions etc, Pross Negotiation through networks, Sebattier Shadow networks, Klickert Chains of accountability, Fine Communication for social change, Rockefeller Wheels and webs, Chapman & Fisher Existing theory www.odi.org.uk/rapid/lessons/theory

9. Existing theory – a short list • Policy narratives, Roe • Systems of Innovation Model, (NSI) • ‘Room for manoeuvre’, Clay & Schaffer • ‘Street level bureaucrats’, Lipsky • Policy as social experiments, Rondene • Policy streams and policy windows, Kingdon • Disjointed Incrementalism, Lindblom • Social Epidemics, Gladwell • The RAPID Framework

10. The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc. The links between policy and research communities – networks, relationships, power, competing discourses, trust, knowledge etc. The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc An Analytical Framework External Influences Socio-economic and cultural influences, donor policies etc

11. Case Studies • Sustainable Livelihoods: The Evolution of DFID Policy • The PRSP Initiative: Research in Multilateral Policy Change • The adoption of Ethical Principles in Humanitarian Aid post Rwanda • Animal Health Care in Kenya: Evidence fails to influence Policy

12. External Influences Campaigning, Lobbying Policy analysis, & research Scientific information exchange & validation A Practical Framework political context Politics and Policymaking Media, Advocacy, Networking Research, learning & thinking evidence links

13. What you need to know • The external environment: Who are the key actors? What is their agenda? How do they influence the political context? • The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? • The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging? • Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?

14. What researchers need to do • Work with them – seek commissions • Strategic opportunism – prepare for known events + resources for others • Get to know the policymakers. • Identify friends and foes. • Prepare for policy opportunities. • Look out for policy windows. • Who are the policymakers? • Is there demand for ideas? • What is the policy process? • Establish credibility • Provide practical solutions • Establish legitimacy. • Present clear options • Use familiar narratives. • Build a reputation • Action-research • Pilot projects to generate legitimacy • Good communication • What is the current theory? • What are the narratives? • How divergent is it? • Build partnerships. • Identify key networkers, mavens and salesmen. • Use informal contacts • Get to know the others • Work through existing networks. • Build coalitions. • Build new policy networks. • Who are the stakeholders? • What networks exist? • Who are the connectors, mavens and salesmen?

15. Policy entrepreneurs Networkers Storytellers Fixers Engineers

16. Practical Tools Overarching Tools - The RAPID Framework - Using the Framework - The Entrepreneurship Questionnaire Context Assessment Tools - Stakeholder Analysis - Forcefield Analysis - Writeshops - Policy Mapping - Political Context Mapping Communication Tools - Communications Strategy - SWOT analysis - Message Design - Making use of the media Research Tools - Case Studies - Episode Studies - Surveys - Bibliometric Analysis - Focus Group Discussion Policy Influence Tools - Influence Mapping & Power Mapping - Lobbying and Advocacy - Campaigning: A Simple Guide - Competency self-assessment

17. Practical Application • Within ODI • Workshops for researchers, policy makers and activists. • Advice to a DFID forest/ground water research project in India: • Less research • More communication • Developing champions in regional and national government • Local, Regional & National advocacy campaign

18. Further Information / Resources • ODI Working Papers • Bridging Research and Policy Book • Meeting series Monograph • Tools for Policy Impact • RAPID Briefing Paper • www.odi.org.uk/rapid

19. Can ILRI do it? Yes, but: • It this its role? • “Global Public Good” Research vs Policy Advocacy • Probably needs to do both: How? • Understand the political context • Get the evidence & package it well • Strategic networking / lobbying / campaigning • Collaboration….

20. ILRI International Livestock Research Institute Process and partnership for pro-poor policy change The New DfID funded Project

21. ILRI International Livestock Research Institute • Project Leaders: ODI / ILRI • Key collaborators: ECAPAPA • Case study collaborators in Kenya: • MoLFD / KARI • Range of NGOs & other SDP partners

22. Why would I be interested? ILRI International Livestock Research Institute • Not all research is expected or intended to lead to policy change, but there may be; • Specific cases where research is expected to; • provide evidence for policy change • identify potential policies (or impact of) • influence the policy making process (advocacy) • Cases where speculative research becomes relevant because of changes in circumstance

23. The project … ILRI International Livestock Research Institute • Ideas for methods and approaches • Lessons learnt from earlier activities • Identification of appropriate communication tools

24. What will we be doing? ILRI International Livestock Research Institute • Three case studies in three DIFFERENT countries • A project considered to have influenced policy change • A stream of research addressing a particular policy area • A clear policy change; • New policy statement • New law • Irrefutable change in way something is done

25. What will we be doing? ILRI International Livestock Research Institute • Three case studies • SDP and impact on changed view of informal milk trade • ???? • ????

26. ILRI International Livestock Research Institute • Discussion: • Can ILRI hope to influence pro-poor policy through research? • Any good case studies?

27. Case Study Approach

28. What is a Case Study? Definition: " A systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" Bromley (1990)

29. Why is it useful? • An ideal methodology when a holistic, in-depth investigation is needed • Designed to bring out the details from the viewpoint of the participants by using multiple sources of data Goal : to describe as accurately as possible the fullest, most complete description of the case.

30. Types of Case Study Types of case studies: • Exploratory, • Explanatory, • Descriptive (Yin, 1993) Stake (1995) included three others: • Intrinsic - when the researcher has an interest in the case; • Instrumental - when the case is used to understand more than what is obvious to the observer; • Collective - when a group of cases is studied.

31. Issues • The unit of analysis is a critical factor • Typically a system of action rather than an individual or group of individuals • Tend to be selective, focusing on one or two issues that are fundamental to understanding the system being examined • Case studies are multi-perspectival analyses • The researcher considers not just the voice and perspective of the actors, but also of the relevant groups of actors and the interaction between them • They give a voice to the powerless and voiceless.

32. Triangulation • Data source triangulation, when the researcher looks for the data to remain the same in different contexts; • Investigator triangulation, when several investigators examine the same phenomenon; • Theory triangulation, when investigators with different view points interpret the same results; and • Methodological triangulation, when one approach is followed by another, to increase confidence in the interpretation.

33. Applications • To explain complex causal links between research and policy • To describe the real-life context in which policy has been influenced by research • To describe the policy influencing process itself • To explore those situations in which the policy intervention being evaluated has no clear set of outcomes.

34. Process • Design the case study protocol: • determine the required skills • develop and review the protocol • Conduct the case study: • prepare for data collection • distribute questionnaire • conduct interviews • Analyze case study evidence: • analytic strategy • Develop conclusions, recommendations, and implications based on the evidence

35. Episode Study Approach

36. What is an Episode Study “A study that focuses on a clear policy change and tracks back to assess what impact research had among the variety of issues that led to the policy change”.

37. What is the purpose? • an excellent way of investigating the influence of research on policy • Can focus on a single episode or comparative episodes. Tracking backwards from policy change to any particular research which influence policy

38. Advantage • The process of working backwards in time gives a more realistic view of the broad range of factors – other than research – that influence policy • Tracking forward probably overemphasizes the importance of research

39. Issues • Policy processes are complex, multi-layered and change over time • Often difficult to isolate the impact of research from other factors • Actors may ‘re-write history’ • Important to seek the views of a wide range of informed stakeholders • The process of preparing an episode study is iterative • Key facts and / or inconsistencies need to be cross-checked with key informants

40. Process • Identify a clear policy change. • Identify key Research Questions (draw on RAPID framework) • Explore how and why those policy decisions and practices took place • Assess the relative role of research in that process by drawing on the framework.

41. External Influences Campaigning, Lobbying Policy analysis, & research Scientific information exchange & validation Apply the RAPID Framework political context Politics and Policymaking Media, Advocacy, Networking Research, learning & thinking evidence links

42. Key Questions • The external environment: Who are the key actors? What is their agenda? How do they influence the political context? • The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? • The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging? • Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?

43. Methods • review of the literature; • interviews with key actors; • capturing the authors’ own experience; and • discussions at workshops. Steps 3 and 4 can be done through a variety of methods:

44. Episode Study Examples

45. International Research The Hubl Study Paravets in Kenya • Professionalisation of Public Services. • Structural Adjustment → Collapse • Paravet projects emerge. • ITDG projects. • Privatisation. • ITDG Paravet network. • Rapid spread in North. • KVB letter (January 1998). • Multistakeholder WSs → new policies. • Still not approved / passed! • Professionalisationof Public Services. • Structural Adjustment • Privatisation • ITDG Paravet networkand change of DVS. • KVB letter(January 1998). • Multistakeholder WSs → new policies. 1970s 1980s 1990s 2000s ITDG projects – collaborative research. Dr Kajume

46. PRSPs – Political Context • Widespread awareness of a “problem” with international development policy in late 90s • Failure of SAPs (and Asian financial crisis) • Mounting public pressure for debt relief • Stagnation of Comprehensive Development Framework idea • Diverging agendas (UK – Poverty, US – Governance) • WB/IMF Annual General Meeting, Sept 1999

47. PRSPs – Evidence • Long-term academic research informing new focus on poverty, participation, ownership, aid effectiveness etc • Applied policy research: • ESAF reviews • HIPC review • SPA Working Groups • NGO research on debt • Uganda’s PEAP

48. PRSPs – Links • WB, IMF, SPA, Bilaterals, NGOs all involved • Formal an informal networks • “None of the players was more than two handshakes away from any of the others”

49. Outcome Mapping

50. What is it? • an integrated PM&E tool • a system to think holistically & strategically about how we intend to achieve result • an approach that focuses on changes in the behaviour, relationships or actions of partners (as outcomes) • a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes • an approach for designing in relation to the broader development context but assessing within your sphere of influence