Program Design and Evaluation A Training Series for Nonprofits Serving Boulder County

1. Program Design and EvaluationA Training Series for Nonprofits Serving Boulder County Session 1: Introduction to Evaluation and Creating a Logic Model

2. Acknowledgements • Boulder County Human Services Strategic Plan: http://buildinglivablecommunities.org/ The Community Foundation, Foothills United Way, Boulder County, The City of Longmont, The City of Boulder, The City of Lafayette OMNI Institute (www.omni.org)

3. Learning Objectives • Describe evaluation basics, including terms and types of evaluation • Depict your planned program work and intended results by constructing a logic model • Understand how a logic model can drive selection of research design and data collection methods

4. Agenda 1. Basics of Evaluation 2. Create a Logic Model 3. Overview of an Evaluation Plan 4. Upcoming Trainings and Technical Assistance

5. Basics of Evaluation Answer this question: When I think about program evaluation I feel….

6. What is Evaluation? Why is it important to YOU ?

7. Evaluation is… …careful collection of information about a program – or some aspect of a program – in order to make necessary decisions about the program. (C. McNamara, 1998)

8. Definition of Evaluation A systematic collection of information about your strategy • Process (activities, characteristics…) • Outcomes (change/impact) That results in • Reduced uncertainty • Improved effectiveness • Informed decision-making

9. Benefits of Evaluation: It’s About Utility • Reflect on progress – where you are now, where you came from, and where to go… • Ensure that program objectives match activities and outcomes • Influence policy makers and funders • Share what does and does not work • Ensure funding sustainability • Strengthen accountability

10. Process Describes the “who,” “what” and “how” of program services. It answers the question, “Is the program being implemented as expected?” Outcome Describes the extent of change (e.g. behaviors or knowledge) as a result of the program. It answers the question, “What was the impact of the program on the clients served?” Types of Evaluation

11. Process and Outcome Activity Worksheet: Process, Outcome and Impact Note: “impact” refers to an intermediate or long-term outcome (e.g. change that occurs more than 12 months following the program and up to 4-6 years out).

12. Steps Taken in Evaluation 1. Create Logic Model 2. Develop an Evaluation Plan 3. Identify an Appropriate Method for your Outcomes 4. Collect & Enter Data 5. Analyze Data 6. Report and Reflect on Data Evaluation is an ongoing process.

13. ‘Would you tell me, please, which way I ought to go from here?’ ‘That depends a good deal on where you want to get to, ’ said the Cat. ‘I don’t much care where’ said Alice. ‘Then it doesn’t matter which way you go,’ said the Cat. - Lewis Carroll

14. How do we get there? Where are we now? Where do we want to be? Logic Model in its Simplest Form

15. What is a Logic Model? • A simplified picture of a program, initiative, or intervention that is a response to a given situation • It shows the logical relationship between the need you will address, the services you will provide, and the changes you expect to see as the result • It is at the core of program planning, evaluation, program management and communication

16. Benefits of a Logic Model • Develops understanding • Helps monitor progress • Serves as an evaluation framework • Helps expose assumptions • Helps restrain over-promising • Promotes communication

17. Creating a Logic Model Problem: Issue or situation to be addressed Services: Activities to change the problem Outcome: Changes or benefits that result Problem / Situation Services Outcomes

18. Creating a Logic Model (Silly!) Example Problem Services Outcomes Take pain reliever pills as directed No more headache! Severe Headache

19. Simplified Logic Model & Terms Problem / Situation Services Outcomes Strategies Activities Outputs Client Characteristics And Needs Outcomes Measurable Indicators

20. Definitions

21. Needs Assessment Choose Program Model (research) Simplified Logic Model Client Characteristics And Needs Strategies Activities Outputs Outcomes Measurable Indicators

22. Overall Theory Base Links the outcome or issue you want to impact to potential causes Program Theory Base Links chosen program to a change in the “causal factors”. Theory: The Missing Link Early Academic Failure School Dropout Research and Evidence Early Intervention (preschool model)

23. Example: School Drop Out (Research Base) “Causal Factors” or “Predictors” (research-based) • Transitions and Mobility • Poverty • Early Academic Failure • Lack of school commitment or “connectedness” Evidence-Based or Research-Based Strategies • Positive Behavior Support • Project ACHIEVE • Positive Action • Perry Preschool Project

24. Theory of Change Discussion Small Groups: • What is an example of research that you utilize to inform your program/strategy? Large group share outs

25. Components of the Boulder County Program Logic Model • Program Goal • Target Population • Program Services • Resources dedicated • Type of service • Extent (or amount) of the service • Impact of the program • Short-term, intermediate and long-term outcomes

26. Program Goal A goal is a broad, generalized statement about what is to be accomplished in the long-term. Goals • Focus on a condition or outcome of interest – based on real community need • Establish the direction and reasoning of a strategy • Avoid defining the problem in terms of a preferred solution • Are framed in a way to build consensus

27. “Good”: To increase early academic success in Alexander school district “Needs Work”: To provide tutoring to at-risk students in Alexander elementary school Goals: Examples

28. Target Population Answers the question: What types of clients will and won’t our program be effective with? Individual characteristics Environmental Variables Abilities Systems Involvement

29. Target Population Also answers the question: What types and level of need will and won’t our program be effective with? Expresses both LevelandType Of Need

30. Example: School Drop Out(Target Population) Example 1: 3-5 year olds with low SES (Socio-Economic Status) in Alexander School District Example 2: 3rd graders not reading at grade-level in Alexander School District

31. Logic Model Activity:Goal Statement and Target Population Individual work • Review/refine or develop goal statement • Review/refine or identify target population Large group share outs

32. Impact of the program Short-term and Intermediate Outcomes • Measures the steps taken that will lead to a change in the long-term outcome such as: knowledge, attitudes, and skills acquired or changed as a result of services Long-term Outcomes (Impact) • Require more time to be achieved and are typically more difficult to connect directly to services due to outside influences that also impact the outcome Client Characteristics And Needs Strategies Activities Outputs Outcomes Measurable Indicators

33. Outcomes and Goals: Definitions Short-Term Outcomes: immediate effects of the program (changes in knowledge, attitudes, skills or level of functioning). Intermediate Outcomes: longer term effects of program that are generally achieved one year after the implementation of the program or strategy (changes in knowledge, attitudes, skills or level of functioning). Long-Term Outcomes or Impact: achieved generally in 4-6 years, and usually show change in behavior, norms, policies, organization and systems. Goals: state the overall purpose of the program, expressed in terms of changes.

34. What Is A Measurable Outcome? • Describes the extent of change (how much) • Identifies the target population(for who) • Specifies the behavior, condition, or knowledge you hope to change (what) • Includes a date or time frame by which the change can be expected (by when)

35. “Good” 25% reduction in early academic failure at Alexander School District within the next 5 years Reduction in early academic failure at Alexander School District within the next 5 years “Needs Work” Encourage school completion Example: School Drop Out (Measurable Long Term Outcome)

36. Example: School Drop Out(Goal and Outcomes) • Short term outcome: All 80 students who receive the Perry Preschool program met at least 90% of the 58 key developmental indicators upon completion. • Intermediate term outcome: 80% of students who participated in Perry Preschool program are on-track academically in the first grade; 80% of students who participated demonstrate less antisocial behavior and misconduct in the first grade. • Long-term outcomes: 25% reduction in early academic failure in Alexander School District within the next 5 years • Goal: Increased early academic success in Alexander School District

37. About Outcomes • There is no right number of outcomes. • Strategies more directly affect short term outcomes than longer term outcomes. • Don’t discard an outcome because it’s affected by other forces. • Outcomes should not go beyond the strategy’s purpose or target population.

38. Logic Model Activity • Develop short, medium or long-term outcome on the logic model. • Considerations: • Is there consistency (logical link) between the short-term, intermediate and long-term outcomes? • Is there consistency between the goal and the long term outcome?

39. Where We Get Confused Getting lost in the timing of outcomes – “Is it an outcome or a short-term outcome or is it really a long-term outcome … no, it’s a goal … I think?” Understanding how our “theory of change” or research-base fit into a model

40. Services and Resources • Type of Service or “Activities”: the processes, tools, events, technology and actions that are part of program implementation (answers the question: what service are we providing?) • Extent of Services or “Outputs”: direct products or deliverables of the program, such as number of classes held, number of hours of the program, number and type of participants, etc… (answers the question, how much of this service are we providing?) • Resources: human, financial, organizational, & community resources (answers the question, with what resources are we providing this service?)

41. Programs/Services Link with: • Target population: needs & characteristics • Program goal(s) and outcomes Quality attributes: • Appropriate for target population • Evidence-based or research-based • Sufficient intensity and duration to achieve expected outcome

42. Examples:Resources, Activities, Outputs Answers the question: What are we providing, how much, and with what resources?

43. Example: School Drop Out (Services and Resources) • Resources dedicated: • Early childhood education teachers, elementary schools, trained coach, evaluation tools. • Type of service: • 1) curriculum, small class sizes and individual-assessments; 2) quality staff, learning environment and quality assessments; and 3) support for low SES families and parent/guardian engagement. • Extent (or amount) of the service: • 80 preschool-aged children with low SES; 36 week curriculum; home visit; 3 CORs per child; 4 PQAs; 2 parent-teacher conferences, monthly newsletters and bi-monthly family gatherings.

44. Review: Creating a Logic Model • Involve program stakeholders • Identify the goals of the program • Specify the target population • Identify short and longer-term outcomes • Outline the services you provide and the resources dedicated to the program • Identify the service amount or outputs

45. Use of Logic Models Once you’ve created a logic model… • Use it to plan your evaluation • How will you measure your outputs? • How will you measure your outcomes? • How will you monitor your implementation? • Review and update it regularly

46. Steps Taken in Evaluation 1. Create Logic Model 2. Develop an Evaluation Plan 3. Identify an Appropriate Method for your Outcomes 4. Collect & Enter Data 5. Analyze Data 6. Report and Reflect on Data Evaluation is an ongoing process.

47. Learning Objectives • Describe evaluation basics, including terms and types of evaluation • Depict your planned program work and intended results by constructing a logic model • Understand how a logic model can drive selection of research designs and data collection

48. Evaluation • Please complete your evaluation of today’s session • THANK YOU!

49. Technical Assistance from OMNI • OMNI trainers will be available on May 29th from 9am to 4pm to meet with individual organizations to answer questions specific to your programs • Sign-up for a one hour slot through The Community Foundation. Registration opens this Friday and is open until all slots are filled • Email EvaluationTA@omni.org and we will respond to specific questions you may have related to the content covered in this training

50. Future Trainings • Selecting Evidence-based Programs, Practices, and Policies • Practical Evaluation Design: Developing a Plan (or How to Get Started) • Practical Evaluation Design: Collecting your Data • Program Refinement: Using your Results