Outcome Evaluation: Logic Models, Evaluation Frameworks, & Measuring Indicators

1. Outcome Evaluation:Logic Models,Evaluation Frameworks,& Measuring Indicators Session #2/3

2. What to Expect from this Series  • Learn about steps required to implement outcome evaluation • Design a program logic model • Identify appropriate evaluation/measurement tools • Develop an implementation plan

3. Position Affiliation Name EMAIL?? Position Affiliation Email Name A note about thenature of evaluation…

4. Purpose of Today’s Session • To complete individualized workable program logic model for program of your choice • To begin designing individualized evaluation plans based on your logic model • To begin identifying indicators and potential evaluation measurement tools

5. Today’s agenda • Answering questions and refining logic models • Break • Analyzing logic models and identifying priority evaluation questions • Lunch • Explaining indicators and measurement • Beginning to draft indicators for your programs • Trying to wrap up by 3 so that we can save time for other questions or one-on-one discussions at the end of the day.

6. Logic ModelsResponding to the Online Exercise • Great start! • Varying stages • Tips What did YOU think of the online exercise?

7. Logic ModelsReactions to your efforts thus far

8. Challenges and Issues • Balancing the logic of program development and promotion with the logic of the program itself • Finding the right balance for short-term and long-term outcomes • Differentiating program process objectives from outcome objectives • Finding outcomes for individualized programs or programs for people in crisis

9. Balancing the logic of program development and promotion with the logic of the program itself

10. Finding the right balance for short-term and long-term outcomes • Try to follow the “logical chain” from top to bottom and look for big logical leaps. Try to identify the unstated assumptions that underlay these leaps. • Look for outcomes that are largely within your control.

11. Finding the right balance for short-term and long-term outcomes Home visits Families experiences are affirmed and normalized ? Families have increased knowledge of local recreation opportunities ? increased recreation and leisure opportunities decreased feelings of depression increased opportunity for participation in community increased acceptance & understanding of accessibility needs of [vulnerable population]

12. Dealing with programs that have individualized activities and outcomes

13. How do we identify outcomes when services are highly individualized? • Look for more immediate short term outcomes that may be common to more clients • Try to categorize the most typical activities and outcomes into broad groups …even the most individualized services tend to have a cluster of typical activities • Choose to emphasize those outcomes that will appeal to key audiences

14. Example of a well-done logic model for a program with individualized outcomes Develop and support clients’ individual rehab plans Provide clinical support and interventions Assess and develop life skills Activities Clients are able to find +/or maintain accommodation of their choice Clients gain insight to healthier lifestyles/choices Clients become aware of +/or are connected to community resources Clients maintain or improve their level of function Short-term outcome objectives Clients have improved informal supports Clients have longer community tenure Clients have improved informal supports Reduce the # of hospitalizations or shorten lengths of stay Long-term outcome objectives Client becomes a more productive, independent member within their community

15. This example works well because… • The outcome objectives are diverse, so they explain the full impact of the program. • The short-term outcome objectives are clear, concrete, achievable and measurable. • The links are differentiated well (i.e., every activity doesn’t link to every outcome). • Certain short-term outcomes emerge as pivotal to the overall logic.

16. How do we identify outcomes when services are crisis response-oriented? • Think “Harm Reduction” or “Secondary Prevention”: Identify the ways in which your intervention reduces negative consequences of crises, or reduces risk of further crisis • Think of the “next steps” your clients can take once stabilized as outcomes • Think of very, very short-term outcomes like, “client feels a sense of dignity”

17. Example of a well-done logic model for a crisis oriented program Weekly self help group for women Activities Women feel supported in the decisions they make Women feel less fragile and are not dealing with daily crises Women develop friendships in the group Women are able to evaluate relationship with partner and make decisions about the relationship Short-term outcome objectives Women support each other outside of the group Partner is involved as part of family decision making Women access resources Women articulate expectations of their partners Long-term outcome objectives Family members reconstruct a support system and maintain family unity

18. This example works well because… • The short-term outcomes don’t imply dramatic change in the lives of participants, which is appropriate for a program dealing with women in near crisis situations. • Even the long-term outcomes are fairly modest and achievable (which is good for the same reason). • The model makes a strong case for having an impact on one aspect of a complex social issue

19. Other Questions?

20. Time to work a bit more on polishing logic models

21. MORNING BREAK

22. Feeding Back • What insights are starting to emerge from your model?

23. Review of Logic Models: Prioritizing Objectivesfor Measurement Discussion Exercise

24. The importance of the arrows in a logic model • The arrows in a model are like your hypotheses. They express your ideas about the cause-and-effect linkages within your program. • Most arrows are based on certain assumptions .. Some are conscious, others are probably not so conscious.

25. Analyzing your Logic Model • If you follow each logical pathway through from activities through short-term outcomes to long-term outcomes, does the logic “hang together” or does it feel like there are unstated assumptions? • Where might data gathering help you to check these assumptions? (make a note)

26. Analyzing your Logic Model • Are there any activities that don’t seem to link to any identified outcomes, or vice versa? • Which logical links do you feel sure are actually occurring in your program? Which ones are you less sure about? • Make a note about those links that you feel you could document better.

27. Analyzing your Logic Model • Which outcomes seem “pivotal” in your model? Which ones are absolutely key to overall success? • Which ones do outsiders least understand or appreciate? • What are the issues or questions that your key stakeholders are going to see as most important? • Make a note

28. Analyzing your Logic Model • Where does your logic model “link up” with the overall priorities of your board, your clients, your funders?

29. Translating key issues or unknowns or assumptions into research questions

30. Clarifying the levels of questions Evaluation Purpose Evaluation Questions Actual Interview or Survey Items

31. Definitions- Outcome Measurement questions - • the questions we need to ask to assess the attainment of each outcome objective • the most central question is most often a rewording of the outcome objective • Other evaluation questions may ask about how the program/activity was implemented, problems encountered, lessons learned, changes made in program implementation, or costs associated with achieving the outcome objective

32. Types of Evaluation Questions Economic Analysis Questions (Are the outcomes worth it?) Outcome Measurement Questions (Have we made a measurable difference?) Satisfaction Questions (Are stakeholders pleased?) Process Questions (Was program implemented as planned? What’s working? What has been learned?) Needs Assessment Questions (What are local needs? local strengths? What are some good ideas for trying to help?)

33. Exercise: Comparing Questions How effective is our program in fostering independence, increasing consumer control, and improving nutrition? What are our strengths and weaknesses? Is our program community based? Does our program improve quality of life? What would our stakeholders like us to change, and what should we keep the same? How successful have we been in involving participants at all levels of the project?

34. A Framework for Outcome Measurement

35. Drafting Evaluation Questions:Exercise • Considering the outcome objectives you chose from your logic model, and; • Considering the stakeholders involved, and; • Considering how you want to use the evaluation results … • What are the core questions that will guide your evaluation?

36. Group Discussion • What are some of the key questions or issues you have decided to focus on? • How did you decide which objectives to focus on? What were some of your criteria?

37. Some critical comments on logic models If we have time ….

38. A logic model… • can come in many different formats. There is no “correct” way to format models although some ways are better than others. As a rule, models are more useful if they are clear about logical linkages and proceed temporally from what you do to what you wish to accomplish. • is not an outcome but a process. You can expect to go through many iterations of your model. • is only useful insofar as it is used.

39. Principles Targets Indicators Inputs Outputs/Benchmarks Implementation Steps Process, implementation, service objectives Short-term Outcomes Long-Term Outcomes Intermediate Outcomes Target Groups Additional Info on process Additional Info on Measurement Goals or Vision Different Approaches to Logic Models Activities

40. Limitations of logic models • Logic models do not do a good job of capturing program context and program process • They can sometimes take on a life of their own. Because they purport to describe the program, people may continue to assume that it is the best rendering of what actually occurs and what the objectives are. • They may contain outcome objectives that are logical but too difficult to evaluate for a variety of reasons. • Raises the question: Should an organization include outcome objectives in their model if they fully expect to be unable to evaluate them?

41. Should PLMs includeoutcome objectives thatcan’t or won’t be measured? • As we move down the logical chain of the program it gets more and more difficult to evaluate our objectives. This is due to many factors including: • time • resources • funding and timing of funding • multiple causal factors outside control of program • Do we still include our longer-term objectives? Activity Attitude Change? Behaviour Change? Societal Change?

42. Group Discussion • What was the hardest part about drafting evaluation questions? • Did thinking about your evaluation questions lead to any insights about your objectives?

43. LUNCH

44. Indicators Objectives and evaluation questions lead to a variety of indicators

45. Definitions:- Indicators - • empirical (observable) bits of information that are used to determine “how we will know” whether the outcome objectives are being achieved • measurable approximations of the outcomes you are trying to achieve Example: Objective: Increased social support among participants Indicator: client perceptions of social support

46. Review:A Good Indicator … is “empirical” in nature • is pragmatic • is as “close” as one can get to the phenomena of interest • helps to further ”operationalize” an objective

47. Review:A Good Indicator … • leads to good measurement and methodology  (e.g., measures degrees of change and not just presence or absence) • is not necessarily quantitative Example: client perceptions of social support

48. Review:Sources of Indicators • Research literature • Data already gathered for other purposes • Observations • Perceptions of key informants • Standardized measurement tools

49. Choosing Indicators: An Exercise

50. The Main Message About Indicators: Good measurement is often an act of creative compromise and balance • no hard and fast rules about how to do them well • strategic combination of 2 or 3 imperfect indicators often works best