Evaluation of Wellness Program Impacts and Outcomes

1. Evaluation of Wellness Program Impacts and Outcomes 1 October 2009 State Agency Wellness Conference Emma Kirkpatrick, MPH

2. Introduction 2 • Why do Evaluation? • Evaluation Basics • Program Goals and Objectives • Logic Models • Designing an Evaluation Plan • Implementing an Evaluation • Evaluation Methods and Tools • Resources

3. Why do Evaluation? 1. An effective evaluation increases the capacity of your program. Comply with grant requirements Can use results to leverage additional funding Documents the work of your program Improves your organization’s ability to track program information 3

4. Why do Evaluation? (cont’d) 2. An effective evaluation can contribute to the broader field of study Establish best practices for interventions Data collection tools can be used and adapted for other projects 4

5. EvaluationBasics Evaluation: a systematic approach to improve and account for programmatic actions A search for empirical evidence to confirm that a program was delivered (process), and to attribute a specific impact on the behavior and health status of a population at risk (outcome). From: Windsor, Clark, Boyd, & Goodman. 2004. Evaluation of Health Promotion, Health Education, and Disease Prevention Programs. 5

6. Evaluation Basics (cont’d) Steps in Evaluation Identify Program Goals and Objectives Define evaluation questions and indicators Define methods Define instruments and tools Implement the evaluation Analyze data and write the report Disseminate and use data 6

7. Evaluation Basics (cont’d) Types of Evaluations: Process (How was the program implemented?) Outcome (Did the program meet it’s objectives?) Impact (Was the ultimate goal of the program achieved?) 7

8. Evaluation Basics (cont’d) Program Goals and Objectives: Goals are the overall purpose of the program – broad, long-term accomplishments Example: The Department of State Health Services will reduce the prevalence of employee smoking by the next fiscal year. Objectives are the steps leading to the goals and the changes required to achieve the goals 8

9. Evaluation Basics (cont’d) SMART Objectives S.M.A.R.T refers to the acronym that describes the key characteristics of meaningful objectives Specific (concrete, detailed, well defined), Measureable (numbers, quantity, comparison), Achievable (feasible, actionable), Realistic (considering resources), and Time-Bound (a defined time line). Example: By the end of the 4-week smoking cessation program, 10% of the participants will have quit smoking.  9

10. Evaluation Basics (cont’d) An Evaluation Logic Model A logic model is a template or framework that outlines program resources, activities, outputs, outcomes,and goals. 10

11. Evaluation Basics (cont’d) Logic Model Example: 11

12. Evaluation Basics (cont’d) Indicators: An indicator is a tool that helps you know how far your program/project is from achieving it’s goals and whether the program/project is headed in the right direction. The right indicator should: Be relevant to the project. Be easily understandable to everyone interested in your project. Be easily measured. Provide reliable information. 12

13. Evaluation Basics (cont’d) Every program has… Goals Objectives Activities 13 Every program evaluation should have… • Impact Indicators • Outcome Indicators • Process Indicators

14. Evaluation Basic’s (cont’d) Goal: To increase the number of employees who consume the daily recommended servings of fruit and vegetables. Objective: By the end of the fiscal year, increase employee access to fresh fruit and vegetables by 10%. Process Indicator: The number of employees enrolled in the Farm to Work program Outcome Indicator: The percentage of employees who report having easy access to fresh fruit and vegetables at the end of the fiscal year as compared baseline Impact Indicator: The percentage of employees who report eating the recommended servings of fruit and vegetables. 14

18. Implementing an Evaluation Quantitative evaluation methods concentrate on systematically measuring changes and effects using numbers and statistics. Surveys are the most commonly used quantitative evaluation tool. Qualitative evaluation methods use words and meanings and focus on describing people's experience and feelings about the program. focus groups in-depth interviews. 18

19. Implementing an Evaluation (cont’d) Evaluation Tools Create an Evaluation Timeline 19

20. Resources Centers for Disease Control and Prevention.  Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11). http://www.cdc.gov/eval/framework.htm Health Improvement: A Comprehensive Guide to Designing, Implementing, and Evaluating Worksite Programs. http://www.businessgrouphealth.org/pdfs/issuebrief_nov2004.pdf Physical Activity Evaluation Handbook. http://www.cdc.gov/nccdphp/dnpa/physical/handbook/pdf/handbook.pdf Wellness Council of America: Free Reports. http://welcoa.org/freeresources/index.php?category=8&PHPSESSID=2bd16514f1dae308134faa8c24cf81b7 Monitoring and Evaluation of Worksite Health Promotion Programs – Current state of knowledge and implications for practice. http://www.who.int/dietphysicalactivity/Engbers-monitoringevaluation.pdf Swift Worksite Assessment & Translation (SWAT) http://www.cdc.gov/nccdphp/dnpa/hwi/program_design/swat/index.htm The Community Tool Box: http://ctb.ku.edu/en/tablecontents/ 20

21. “Accountability breeds response-ability.” - Stephen R. Covey 21