Outcome Mapping: A Method for Measuring the Results of Social Change Interventions

1. Outcome Mapping:A Method for Measuring the Results of Social Change Interventions Child Rights in Practice: ‘Measuring Our Impact’ May 10-15, 2009 Whistler, British Columbia, Canada Terry Smutylo tsmutylo@magma.ca

2. Two approaches to Evaluation: • As inspection: • Judge success or failure • Measure against fixed goals • External for objectivity • Linear cause/effect models • Accountability to external • Accountability for control, blame • Evaluator controls evaluation • Engender fear of failure • As supporting innovation: • Provide feedback for improvement • Adapt measurements as goals evolve • Internal, integrated, interpretive • Seek to capture system dynamics • Accountability to values, commitments • Understand & respond strategically • Evaluator matches process to context • Feed hunger for learning Adapted from: Patton, Michael Q., 2006, “Evaluation for the Way We Work”, The Nonprofit Quarterly, Spring.

4. An evaluation conundrum the solution… changed behavior program influence decreases community capacity & ownership increases miracles occur

5. Challenges in measuring « results » Change is: • Complex (multiple actors and factors) • Continuous (not limited to the life of the project) • Non-linear (often unexpected, emergent) • Two-way (program may also change) • Beyond the control of the project • Incremental, cumulative (watersheds & tipping points)

6. key ideas in

7. ...Focus on direct partners The individuals, groups, and organizations you work with directly to support change

8. …Influence, not «impact» • The project contributes to a better world • …by influencing the behaviour (relationships, interactions and or actions) of its partners

9. Pay attention to what happens along the way… The journey is as important as the destination.

10. a brief history • mid-1990s: need to demonstrate results • 1998: Barry Kibel and Outcome Engineering • 1999: methodological collaboration with FRAO & NEPED (IDRC funded projects) • 2000: publication of manual in English • training, facilitation & usage globally • 2006: www.outcomemapping.ca • 2008: CLAMA

11. What’s new about OM? • “Outcomes” = changes in behavior • Focus on actors who will drive & maintain the changes – “Boundary Partners” • Sets out the pathways of change with – “Progress Markers” • Recognizes the boundaries of the project’s influence

12. Beneficiaries Partners Project Identifying system boundaries sphere of interest sphere of influence sphere of ‘control’ Adapted from: Steff Deprez VVOB-CEGO, Nov 2006

13. What are we trying to accomplish and how? What do we want to track & how? What do we want to learn & report?

14. Intentional Design step 1: vision

15. Vision improved human, social, & environmental wellbeing

16. I have a dream! Martin Luther King, Jr. August 28, 1963

17. vision guides… …and energizes the team

18. Swayamsiddha Vision Across rural India, women and girls utilize and benefit from appropriate health care, education, food and water security and freedom from violence. Women have access to the markets, credit, banking and municipal services they need to pursue their livelihood goals. They use drudgery-reducing technologies and agricultural inputs that contribute to personal well-being and to ecological sustainability. Villages are fully served by public transport, are well lit at night and police enforce all laws fully and equitably. Girls attend school full time and families have the information and resources to make informed decisions regarding their health, safety and social needs. Gender equity governs household labor and decision-making; and men in the community understand and support gender-responsive laws. As elected representatives and as pressure groups, women influence decisions and policies affecting the wellbeing of their families and communities.

19. vision in graphic form, Nagaland (India)

20. Intentional Design step 2: mission

21. mission statement Describes how the program intends to: • Apply its resources in support of the vision • Support the achievement of outcomes by its direct partners (what it does, with who, how and why)

22. Example Mission Statement The Swayamsiddha Project works with governments, NGOs & CBOs to improve women’s health and empowerment. It facilitates the formation of women’s self help groups by providing funding and training to help them influence community and government services to be more responsive to their health and livelihood needs. The project fosters mutual respect and joint action between women’s self-help groups and banks, police, health & social service providers and government agencies. It researches and promotes ecosystem and human health practices in agriculture and in community health and sanitation services. Swayamsiddha addresses equity issues in all its activities. It uses participatory methods to monitor progress, to learn how to become more effective in supporting its partners and to report on its results.

23. Summary Vision Mission • about the future • observable • idealistic • not about the program • feasible • identifies activities and relationships • about the program

24. Intentional Design step 3: boundary partners

25. sphere of influence The part of the world you are interested in Program = the project’s direct or ‘boundary’ partners

26. moving from stakeholders...

27. ...to boundary partners project

28. Intentional Design step 4: outcome challenge

29. what indicators do you use for: Greater awareness… Empowered women… Community ownership… Reduced conflict… Increased collaboration… Governmental commitment… Gender sensitivity… Equal access… Budgetary transparency… Active participation… Poverty alleviation… Strengthened capacity… ?

30. outcome challenge statement • Describes the behaviours… • of a single boundary partner… • contributing maximally to the vision

31. Intentional Design step 5: progress markers

32. progress markers (ladder of change) Love to see Expanding influence, helping others, sharing expertise Like to see Actively engaged, learning, commitment Expect to see Early encouraging response to program, initial engagement

33. Progress Markers for local communities • Participating in regular model forest meetings • Establishing a structure for cooperation • Acquiring new skills for managing model forests • Contributing resources to get the MF operational • Articulating a locally relevant vision for the MF • Promoting their MF nationally • Expanding the partnership • Calling upon external experts for advice • Requesting new opportunities for training • Publishing examples of benefits achieved through MF • Seeking out new partners for the MF • Obtaining funding from different national sources • Helping other communities establish MFs • Sharing lessons learned internationally • Influencing national policy debates on resource use

34. progress markers: • stimulate the program to try to contribute to profound transformations • provide basis for dialogue with partners • articulate the complexity of the change the program is engaged in • facilitate ongoing assessment of progress • signal need for mid-course corrections and improvements

35. mission vision Banks Girls & Women Women’s Self Help Groups Police BAIF State NGOs Families Community Leaders Public Health Clinics Project’s Outcomes Strategies StrategicPartners BoundaryPartners BP’s outcomes

36. Intentional Design step 6: strategy maps

37. 6 kinds of strategies

39. ? what networks or relationships will be established or utilized? how will sustained support, guidance or mentoring be provided? E aimed at boundary partner’s environment what will be done to produce and “immediate output”? what will be done to alter the physical or policy environment? causal persuasive supportive how will the media or publications be used? I aimed atindividual boundary partner what will be done to build capacity? strategies facilitation questions I-3 I-1 I-2 E-1 E-2 E-3

40. value of strategy maps • Clarify what the project is doing, with whom and why • Anticipate the program’s influence on the BP • Articulate the range, mix and logic of the strategies • Detect strategy gaps and over extension of resources • Encourage multiple interventions to increase effectiveness • Suggest appropriate evaluation methods

41. OM principles of use: Flexible: modular to be adapted to use & context Complementary: combine with other methods

42. OM principles of use Participatory: in dialogue and collaboration with partners Evaluative: promote culture of reflection, results oriented thinking, and social & organizational learning

43. Points to Remember: • Stay in touch with the complex reality. 2.Act out of passion, integrity & competence 3.Celebrate the contributions of your team and your partners. 4.Be “idealistic realists.” 5.Learn, then teach upwards.

44. for more information and to share experiences www.outcomemapping.ca www.idrc.ca/evaluation www.mapeodealcances.net

45. What is “monitoring”? …routine, ongoing, collection of information on a program’s outputs, outcomes and indicators to measure, improve and report on the performance.

46. What is “evaluation”? …the systematic acquisition and assessment of information to provide useful feedback about a program, policy, technology or activity.

47. M&E – what’s the difference? …What’s the relationship?

48. uses of M&E • Feeding learning into management cycle • Meeting accountability / reporting requirements • Reducing risk • Helping a partner learn • Decision-making • Inform publicity documents, communication activities, or case-study materials • What else…?

49. Program Partner situational data Strategies outcomes (behaviour changes in the partners) relevance & viability (of the program) implementation (interventions by the program) 5 kinds of M&E information c o n t e x t u a l i n f o r m a t i o n

50. Useful M&E Has 4 Dimensions ‘Rhythm & Spaces’ (Info to users/events/audiences) Clarifying Intent (Of interventions & outcomes) M&E Implementation (Data collection & interpretation) M&E Planning (Choosing what to track & how)