1 / 83

Partnership

Partnership. What to Expect From This Series. Learn about steps required to implement outcome evaluation Design a program logic model Identify appropriate evaluation/measurement tools Develop an implementation plan. Understanding Outcome Evaluation: Core Concepts and Building the Foundations.

ardice
Télécharger la présentation

Partnership

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Partnership

  2. What to Expect From This Series • Learn about steps required to implement outcome evaluation • Design a program logic model • Identify appropriate evaluation/measurement tools • Develop an implementation plan

  3. UnderstandingOutcome Evaluation:Core Concepts andBuilding the Foundations Session #1/3

  4. Purpose of Today’s Session • To gain a basic understanding of the core concepts and basic steps in outcome measurement in particular, and program evaluation in general. • To share strategies for building buy-in to outcome measurement. • To begin work on a program logic model for a program of your choice.

  5. Today’s Agenda • Outcome evaluation language • Outcome evaluation steps • Break • Outcome evaluation steps • Lunch • Developing logic models

  6. Our Approach toOutcome Measurement

  7. Discussion Imagine…

  8. Our Approach toOutcome Measurement • Evaluation is already going on, informally, in any healthy organization and can be harnessed. • Evaluation is a natural and necessary part of organizational development, not an “add on.” • Evaluation strategies that help one organization may be confusing or even damaging to others: there is no “cookie cutter” approach.

  9. Our Approach toOutcome Measurement • Developing common language within and between organizations is key. • Organizations get better at measuring outcomes when they have practice, and when they identify measurement priorities themselves. • Evaluation is about organizing, interpreting and acting on information, not about gathering mountains of data.

  10. Improved Outcome Measurement Useful Insights, New Questions ClearEvidenceof Impact ImprovedPractice

  11. A Few Key Definitions Community Based Research is a term that refers to a variety of efforts to apply research tools and strategies to the task of building communities. Community based research can take many forms, including needs assessment, environmental scans, etc. .. Program evaluation is one form of community based research. It focuses on the study of community interventions...Their processes, outcomes, and context. Outcome evaluation is one type of program evaluation,in which: 1. outcome objectives for activities are identified,2. success in reaching objectives is measured, and3. best practices are identified, and4. recommendations for improvement result. Outcome measurement is a process in which:1. outcome objectives are identified and2. success in reaching objectivesis measured

  12. The Dilemma of Terminology When TalkingAbout… Evaluators and Funders Use Terms Like… The Process of Measuring and Learning from Outcomes • Stakeholder Consultation, Participatory Approach,Utilization Plan, Lessons Learned, Implementation, Process, Evaluability The Intent of an Intervention • Outcome Objectives,Implementation Objectives, Goals, Outcomes, Targets, Intended Results The Changes Caused by an Intervention • (Short and Long Term) Outcome Objectives,Goals, Impacts, Results The Information Used to Measure Change • Inputs, Outputs, (Performance)Indicators,Targets, Benchmarks, Data Sources

  13. Outcome Measurement The Basic Steps

  14. The Seven Practical Steps in Outcome Measurement 1. Get Buy-In 2. Clarify Theory 6. Communicate Results 7. Act on Results 3. Develop Evaluation Purpose & EvaluationQuestions Laying the Foundations Evaluation Planning Acting on Findings Gathering & Analyzing Information 4. Design Methods and Measurement plans 5. Analysis Plan

  15. Data Doesn’t Change the World… People Do. Evaluation Requires an Ongoing Focus onRelationship Building

  16. NEW CONNECTIONS TO OTHER PEOPLE OR GROUPS NEW INSIGHTS INTO ISSUES The Role of Collaboration and Teamwork in Community Based Research Situating Yourself: Clarifying Values & Beliefs, Reviewing Resources Clarifying The Issues and Identifying Research Questions Deciding & Acting Together Using Knowledge Building & Maintaining Relationships For Action NEW SKILLS OR CAPACITIES Gathering, Sharing, and Organizing Knowledge

  17. Step 1:Building Stakeholder Involvementfrom the Beginning

  18. Fostering Buy-in/Stakeholder Participation: How? • Have all those who have a stake in the project and its evaluation been identified? Is there support for the idea of evaluation? • Are there other ways that stakeholders could be included in the early planning phase and throughout the evaluation? • Has the political context of relationships among stakeholders been considered? • Have the interests of outside audiences been identified and considered?

  19. Creative Ways toInvolve Stakeholders • Helping to build logic models • “Teaching” the evaluators • As surveyors or interviewers • As pilot testers for data collection tools • As note-takers at focus groups • As co-presenters of findings • As participants in a “data analysis forum” • OTHERS?

  20. Discussion • The first meeting of your evaluation committee • In the context of measuring outcomes, when do you run into stakeholder conflict?

  21. BREAK

  22. Step 2: Clarifying YourTheory of Change

  23. What is aTheory of Change? • A way of clearly explaining why you do what you do • An explanation that focuses on exactly howyour work will lead to positive change.

  24. Discussion Which one of the followingtheory statementsbest describesprogram impact?

  25. We have implemented a new type of women’s shelter… • because a local report identified the need for more shelter beds locally. • so that our organization can provide shelter for 20% more women • that is at a different location more accessible to younger women • so that young women will be more likely to report abuse and seek support • in order to identify instances of abusive behaviour earlier in relationships and increase the chances of breaking the cycle of abuse

  26. Components of aBasic Program Logic Model • Activitiesexplain the concrete things you do within your program • Short-Term Outcome Objectivesidentify the fairly immediate changes you expect to see as a direct result of your work • Long-Term Outcome Objectivesidentify the more distant benefits or changes you hope to contribute to by achieving your short-term outcomes • Goalsare the longest term, broad vision of your program

  27. The Really Useful Idea(articulating your theory of change) Identify short-term outcome objectives that are within your control, measurable and achievable, and use a logic modelto explain how these short-term achievements contribute to the achievements of long-term outcomes over time.

  28. Refer clients to Job Leaders Activity Improve clients’ knowledge of job opportunities Short-termOutcome Improve clients’ employment networks Long-termOutcome

  29. How Do Logic ModelsHelp You to Measure Outcomes? • They help “unpack” complex, important long-term outcomes into short-term outcomes that can more easily be measured • They help you sort out exactly how and why you expect outcomes to occur, so you know where to look for evidence of success • They help you distinguish actual “measurement problems” from the much more common “theory problems”

  30. Components of aBasic Program Logic Model(After Rush & Ogborn (1991) CJPE) Activity Clusters • The concrete things you do within your program • The major categories or clusters which describe the activities that make up your work • Clusters should be composed of activities that are similar at a behavioural level, not activities with similar purposes (e.g., cooking in the kitchen and cooking at the barbecue are similar activities, but cooking dinner and buying the groceries for dinner are not).

  31. Components of aBasic Program Logic Model Outcome Objectives • Measurable in principle • Describes a meaningful change • Need not specify how much change will occur or what the target level is • Specifies a particular population and situation • Clearly explained and theoretically defensible • Focused on active rather than the neutral or passive change (i.e., “improved”, or “reduced” vs. “supported”, “promoted”, or “encouraged”) Example: Increased feelings of social support among participating parents

  32. Components of aBasic Program Logic Model Short-term Outcome Objectives • The immediate benefits or changes that the target groups are anticipated to experience or display as a result of the program activities. • It should be possible to identify a clear, direct intended causal link between at least one program activity and each identified short-term outcome objective.

  33. Components of aBasic Program Logic Model Long-Term Outcome Objectives • The more distant benefits or changes that the target groups are anticipated to experience or display as a result of the initiative. • Generally, long-term outcome objectives are the second-order changes that result from successful achievement of short-term outcomes over time.

  34. Refer clients to Job Leaders Improve clients’ knowledge of job opportunities Improve clients’ employment networks

  35. Example of a well-done logic model for a program with individualized outcomes Develop and support clients’ individual rehab plans Provide clinical support and interventions Assess and develop life skills Activities Clients are able to find +/or maintain accommodation of their choice Clients gain insight to healthier lifestyles/choices Clients become aware of +/or are connected to community resources Clients maintain or improve their level of function Short-term outcome objectives Clients have improved informal supports Clients have longer community tenure Clients have increased confidence in ability to live independently Reduce the # of hospitalizations or shorten lengths of stay Long-term outcome objectives Client becomes a more productive, independent member within their community

  36. Example of a well-done logic model for a crisis oriented program Weekly self help group for women Activities Women feel supported in the decisions they make Women feel less fragile and are not dealing with daily crises Women develop friendships in the group Women are able to evaluate relationship with partner and make decisions about the relationship Short-term outcome objectives Women support each other outside of the group Partner is involved as part of family decision making Women access resources Women articulate expectations of their partners Long-term outcome objectives Family members reconstruct a support system and maintain family unity

  37. Working on Logic Models2-4 pm

  38. Exercise 1: Describing your Activities • Think about what your staff and/or volunteers do on a day to day basis in as concrete terms as possible. • Focus on activities that involve contact with clients or the community. Activities that should be in your model are those that you expect will impact individuals or communities in some positive way. • Staff are most often “doing things”, like “providing”, “teaching”, “raising awareness”, “creating”, etc. so try to use these types of active verbs. List as many activities as you can and try to be as honest as you can about what you actually do. • Try and “cluster” your activities if they seem to involve doing similar things.

  39. Avoid putting “outcome” language into your activities. For example, avoid terms like “prevent”, “increase”, or “improve” because these are changes that are a consequence of the activities. Identify the things your program does that leads to prevention, increases, or improvements. • Avoid “double barreled” activities (e.g., “provide education and personal support”). This is especially important if they might lead to different outcomes. Separate them if possible. • While administrative tasks are important and vital (e.g., team meetings, hiring functions, etc.), you can usually omit them in logic models because they do not tend to directly impact outcomes objectives. You can leave these out and add them later if it helps to clarify your program. Exercise 1: Describing your Activities

  40. Exercise 2: Identifying the Outcome Objectives for your Program • Develop a list of the outcome objectives your program is intended to achieve. Include as many outcomes as you can. You might also think about your mission, your marketing materials, grant proposals, and your own reasons for doing your job. • Focus on how the program makes a difference or change in your clients or the community. For example, if an outcome objective of the program is to give people someone to talk to, ask yourself how this may help an individual. • Try to write your outcome objectives in way that refers to change that can be measured. • Think about sequence. Logically, some outcome objectives will necessarily precede others. Try to arrange them temporally from top to bottom.

  41. Exercise 3: Putting it all Together • Using a pencil, begin to draw the connections between your activities and your short term outcome objectives and between your STOs and your long-term outcome objectives. Keep in mind the following: • All STOs should be linked to at least one activity; all activities should be linked to at least one STO. • All LTOs should be linked to at least one STO; all STOs should be linked to at least one LTO. • Pick an arrow that you think might have questionable logic or weak assumptions and make it “dashed”. • Pick an arrow that you think has strong logic and well-supported assumptions and put a check mark beside it . • Think about logical causes. If a group support intervention is supposed to lead to greater feelings of social support, think critically about why and how that happens. Does it make sense? In some cases, this analysis might lead you to add, delete, or modify activities and outcome objectives.

  42. Using theOn-LineLogic Model Design Tool

  43. What to Expect at Our Next Session • Respond to questions regarding finishing up logic models • Identify indicators • Start thinking about measurement tools

  44. Upcoming Workshops • Workshop 2 focuses on • Refining logic models and identifying measurement implications • Workshop 3 focuses on • Translating measurement plans into action

  45. Thank You!

  46. Step 3:Developing YourEvaluation Purpose &Evaluation Questions What do you really want to know? What kinds of questions canevaluation really answer for you?

  47. Breaking Down the Complexity What were we trying to change? (Long-Term Outcome Objectives) What particular contribution were we going to make to that change? (Short-Term Outcome Objectives) Did our program have an impact??? How were we planning to make that contribution? (Activities) What sorts of things would we see if the expected change was happening? (Indicators) How can we document those observations in a systematic way? (Methods)

  48. The Evaluation Purpose Statement • A purpose statement should specify exactly what the goals of the research project are, and what you intend to investigate in order to achieve these goals.

More Related