Healthy Schools Leadership Program (HSLP) Evaluation Plan

1. Healthy Schools Leadership Program (HSLP)Evaluation Plan February 2006

2. What I’m going to talk about • Present & get your feedback on Evaluation Plan for HSLP 06-08 cohort • Generate ideas for your own assessment/evaluation work • Ideas for using the Healthy Youth Survey  Please consider this a discussion!

3. “George tried to be a good monkey…”

4. The Primary Evaluation Question • Is the Healthy Schools Leadership Program (HSLP) effective for making schools healthier? • Note that this is a state-level evaluation question • I will present an ambitious scenario for evaluation • We will attempt to be as rigorous as possible • I’m not a schools expert… you are the schools experts

5. Caveats to Evaluation Model • Small numbers of schools (n=15) • Change takes time – especially for long-term outcomes • Our comparison group is the rest of the state • If findings are positive, some will argue that this is the results of your existing capacity/readiness • The plan is ambitious – looking for additional resources to support evaluation

6. I’d like to say something stronger What we can say right now with certainty is that if you spend resources implementing coordinated school health it doesn’t appear to make your academic achievement any worse….

7. Describe the Program

8. Coordinated School Health Goals • Health Education • Physical Education • Counseling/Support Services • Food/Nutrition Services • Healthy Environment • Health Services • Family/Community Involvement • Staff Wellness

9. Simplified Logic Model A. Inputs Teams HSLP training B. Activities Ass’t Plan Evaluate C. Outputs Ass’t results Action Plan D. Short-term Outcomes Results from action plan activity E. Intermediate Outcomes Policy, procedure, environment changes Sustainability F. Long-term Outcomes Staff/studn’t behavior change G. ImpactsHealth improvem’t Academic achievem’t Catalysts Barriers Non-HSLP External Factors

10. Getting the Right Perspective: Evaluation Questions

11. Specific Aim 1. Describe the extent to which the inputs and activities are sufficient to generate outputs – assessment results, action plans (A/BC, Descriptive) • Were teams able to assemble wellness committees? How did they function? Were they able to complete assessment and planning? What priorities were identified? What were the barriers and motivating factors related to output completion? • Are there differences among HSLP teams in satisfaction with outputs? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in achievements of high-quality outputs? Data Sources: Team interviews, action plans, Healthy Schools Report Card

12. Hypothetical Findings • All schools assembled teams; X% included community members • 12 of 15 teams created formal action plans • All schools identified at least one nutrition priority • School teams that were “very satisfied” with their plans were more likely to have involved community members and had a clear process for translating assessment results into an action plan

13. Specific Aim 2. Describe the extent to which participation in the HSLP is associated with short-term outcomes – early results of action plans (A/BD, Descriptive) • Were school wellness teams able to use plans to initiate change? How did implementation activity begin? Which priorities did schools take action on first? How were responsibilities identified? What factors were associated with transition from planning to action? • Are there differences among HSLP schools in their satisfaction with/quality of early implementation? Were some kinds of activities more easily translated to quality action? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in initiating high-quality work? Data Sources: Team interviews, action plan reports, Healthy Schools Report Card

14. Hypothetical Findings • 10 of 15 schools increased the overall number of “maintenance” items on their Healthy Schools Report Card • Teams reported highest satisfaction with their work on XXX priorities • XX Teams (X%) said they “strongly agreed” that change would not have occurred without the HSLP training and technical assistance

15. Specific Aim 3. Describe the extent to which schools that participated in HSLP change their environments (A/BE, Pre/post comparison to non-HSLP schools) • Did HSLP school environments change, and were changes different than those of non-participating schools? Which changes were achieved? • Are there differences among HSLP schools that achieved policy/procedure/environment change vs. those that did not? • How has sustainability been addressed? What additional resources have been attracted for school health? Has institutionalization of school health work been achieved? Data Sources: SHEP, Healthy Schools Report Card, HYS (perceived school environments), Staff wellness survey (if perceived environment indicators included)

16. Sample Findings Source: SHEP 2004

17. Specific Aim 4. Describe the extent to which schools that participated in HSLP change their student/staff behaviors (A/B  F, Pre/post comparison to non-HSLP schools for students, self-comparison for staff) • Did HSLP schools’ student behaviors change, and were changes different than those of non-participating schools? • Did HSLP schools’ staff behaviors change? • Are there differences among HSLP schools that achieved behavior change vs. those that did not? Data Sources: HYS, Staff Wellness Survey

18. Sample Findings Source: Healthy Youth Survey 2002-2004

19. Specific Aim 5. Describe the extent to which schools that participated in HSLP have had health impact (A/B  G, Pre/post comparison to non-HSLP schools for WASL, self-comparison for absenteeism) • Did HSLP schools’ student academic achievement change, and were changes different than those of non-participating schools? • Did HSLP schools’ student and/or staff absenteeism rates change? • Are there differences among HSLP schools that achieved impact vs. those that did not? Data Sources: WASL, Student absenteeism measures, Staff absenteeism measures

20. Sample Findings

21. Data Sources: Gathering Credible Evidence “Not everything that counts can be counted; and not everything that can be counted counts.” — Albert Einstein

22. Data Sources • Proposing to use data you already collect, or new data collection related to your HSLP implementation • Exception: interviews of team/administration • For anything where we want to look at everyone together, methods have to be standard… • For some things it would be fine to have sentinel schools • See school-specific data suggestions later

23. Team Interviews • During Spring 06 develop interview protocols for assessing function and satisfaction among HSLP participants (see evaluation questions) • Repeat in Spring 08 • Augment with interviews with other wellness committee members (or lead administrators?) • Your role: participate in interviews

24. Healthy Schools Report Card • Conduct assessment to identify priorities for action planning • ASCD (Association for Supervision and Curriculum Development) has a clear process/instrument • Your role: use good assessment to develop plans, provide a copy of whatever baseline (spring 06) and follow-up (spring 08) assessment results

25. HSRC Component Areas • School health program policy & strategic planning • Coordination of school health programs • Social and emotional climate • Family and community involvement • School facilities and transportation • Health education • Physical education and physical activity • Food and nutrition services • School health services • Counseling, psychological and social work services • School-site health promotion for staff

26. HSRC Indicator Priority Rankings • Multiple specific topics within components • Rate with your team for each: current status, perceived benefit, perceived effort • Color-coded result report for combined scores • Green = short-term high priority • Red = short-term lower priority • Lime = long-term, highest priority • Blue = long-term, high priority • Orange = long-term, lower priority • No color = Maintenance

27. HSRC Sample Component-Topic • School health program policy & strategic planning • All staff members are provided with time/resources to comply with health program policies • Coordination of school health programs • School health program members regularly inform the principal, district superintendent, or school board of current developments in the school health program.

28. Annual Action Plans & Reports • Design your action plan based on assessment results • Your role: provide action plan annually, report of progress annually (format?)

29. School Health Education Profile (SHEP) • Survey of Secondary School Principals and Lead Health Teachers • Sponsored by CDC • Will be sent in Spring 06 and Spring 08 • Your role: encourage completion of the survey

30. SHEP/Profiles Data Examples:School Health Advisory Groups • Have a School Health Advisory Group • 59% of high schools • 56% of middle schools • 46% of combined secondary schools • 55% overall • Similar to 52% total in 2002

31. SHEP/Profiles Data Examples:Tobacco • A designated individual has primary responsibility for seeing that the tobacco prevention policy is enforced • 64% high school • 68% middle school • 59% secondary combined • 65% overall • Similar to 68% overall in 2002 • No association with school advisory presence

32. SHEP/Profiles Data Examples:Nutrition • Students can purchase fresh fruits/vegetables from vending machines or at the school store, canteen, or snack bar • 49% high school • 45% middle school • 39% secondary combined • 46% overall • Slight improvement from 39% overall in 2002 • No association with school advisory presence

33. SHEP/Profiles Data Examples:Asthma • Obtain and use an Asthma Action Plan (or Individualized Health Plan) for all students with asthma • 77% high school • 77% middle school • 74% secondary combined • 76% overall • Improvement from 62% overall in 2002 • Schools with advisories are twice as likely to obtain asthma plans as schools without advisories

34. Staff Wellness Survey • Designed by CDC – available in March • Assuming we can modify (shorten) if needed • Assuming we (DOH/OSPI) can provide as online survey and give data back to school teams • Are there other ways to measure staff absences? • Your role: engage staff to participate • Caution: poor participation will not yield useful data!

35. Healthy Youth Survey (HYS) • Administered Fall of even years • Grades 6,8,10,12 • How many HSLP schools participated in 2002, 2004? • Most powerful comparison will be 02-04-06 • Your role: register to participate in 2006 (soon!)

36. Student Absenteeism • NCLB requires measurement of excused/unexcused absences • Assuming no change in non-illness excused absences, we can monitor changes in this rate over time • Your role: provide annual absence rates

37. Academic Achievement/WASL • Annual, building-level • Grades 4,7,10 • Math, Reading, Writing • Your role: none needed

38. Other School Factors • Free & reduced lunch enrollment [this may be an outcome, consider different measure of SES] • Percent minority enrollment • School/district size/enrollment • School level (elementary, middle, high, combination) • Urban/rural location • Baseline levels of outcomes • Other (baseline) funding sources for health improvement • Your role: none – we can obtain

39. Too much information?

40. Timeline

41. School-level Evaluation • We won’t evaluate you as individual teams • Evaluating yourself on a few points would be helpful in telling your story later • Suggest designating a lead for local evaluation (just like any implementation piece)

42. Sample School-level Evaluation Data • vending revenue data • key informant interviews with students or staff • tracking of policy enforcement activities • other data gathering from communities • health room utilization data • student absences – use more advanced tracking system (Public Health Tracking System grant available from DOH environmental health) • lots of (digital) pictures of your environment – now and later

43. Still Awake??

44. Healthy Youth Survey

45. Why do I need HYS now? • Contribute to assessment, planning • Identification of your priorities • Describing affected students • Gain attention for student health as a priority

46. HYS 2004: Collaborative Effort of OSPI, DOH, DSHS, CTED, FPC, and RMC Research • State-level simple random sample of schools • County-level samples drawn as appropriate • Non-sampled schools also invited to participate • Survey consistently administered in the Fall of even years • Survey given to 6th, 8th, 10th, and 12th graders • Survey booklets have one-page tear-off answer sheet • Survey for 8th, 10th, and 12th graders uses 2 form “interleaved” administration

47. HYS 2004: Participation • The Healthy Youth Survey 2004 was completed… By 185,095 students… In 1,013 schools… In 235 school districts… In all 39 counties.

48. Risk & Protective Factor DataConsistent with CTC & MTF SurveysDevelopment led by DSHS/DASA & OSPI • WSSAHB Questionnaire • Demographics • Alcohol, Tobacco, other drugs • School risk/protective factors • Community risk/protective factors • Peer-Individual-Family risk/ protective factors

49. Youth Health Behavior DataConsistent with National YRBS & YTSDevelopment led by DOH • YRBS Questionnaire • Demographics • Alcohol, Tobacco, Other drugs • Nutrition • Physical Activity • Safety Behaviors (Helmets, Seatbelts) • Mental Health/Depression/Suicide • Additional Tobacco Indicators

50. Getting the Best of Both Form A WSSAHB-like Form B YRBS-like Survey Core