Outcomes Assessment in School Mental Health

1. Outcomes Assessment in School Mental Health Webinar and dialogue sponsored by the Quality and Evidence-Based Practice (QEBP) Practice Group and the IDEA Partnership March 16, 2011

2. Today’s webinar • Overview of QEBP practice group • Highlights of QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM) • Evaluation of school mental health (SMH) programs in Minnesota Public Schools (Dr. Mark Sander) • Psychosocial, Placement and Cost Assessment in Maryland’s SMH programs for youth in special education (Dr. Sharon Stephan) • QEBP updates from the CSMH (Dr. Nicole Evangelista) • Call for Papers on SMH outcomes assessment - special issue of Advances in School Mental Health Promotion(Dr. Michael Kelly)

3. QEBP Practice Group Mission and Priorities • Mission: To provide resources and promote sharing of information across individuals and groups interested in improving the quality of school mental health (SMH) programs and services. • Priority areas include: • improving dissemination and sharing of evidence-based practices in SMH; • bridging the research-practice and practice-research gaps in the field; • and, understanding and promoting the use of the best student-, program-, and school-level evaluation strategies.

4. QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM) • Split into two groups (research and front line) to answer 3 questions: • What are our most pressing concerns related to Quality and Evidence-based Practice? • What is one specific action sep for our group (research or front line) this coming year? • What is one specific action step you would like to see from the “other” group this coming year?

5. Next Steps for QEBP Practice Group • Future webinars and dialogue

6. Mark Sander, Psy.D., LP Hennepin County/Minneapolis Public Schools mark.sander@co.hennepin.mn.us 612-668-5489 School Mental Health: Local Evaluation and Research Lessons Learned

7. School Mental Health Programs Across MN • Minneapolis Public Schools established in 2005 in five schools • Gradually expanded to 15 schools • County wide – 10 agencies, 14 school districts, over 70 schools • State wide – 21 grantees, serving 63 counties, about 200 school districts, and 550 schools

8. Collaboratively Building a Evaluation Model • What is “must have” data? • Mental Health Outcome (SDQ and CASII) • Educational Outcome (Suspension and Attendance) • Access and Engagement • Who informs it? • Stakeholder “interviews” • Going back and talking to stakeholders • Survey of Principal, Assistant Principals and Social Workers (later years)

9. Data Informed Sustainability Planning How do you do it? We went low tech for the first 4 years Now we have developed on-line database How do you fund it? Used SS/HS funding to develop database and work with statistician on analysis How do you use it? Developed preliminary evaluation model Referral and service trends Impact on programming decision Impact on financial model and cash flow for agencies

11. Utilizing the Data to Inform and Sustain Referral and Service Trends Impact on programming decision Impact on financial model and cash flow for agencies Case for Ancillary and Supportive Services Teacher Consultation Care Coordination Training for educational staff; classroom presentations Case for educational Benefit Reduction in Suspensions Reduction in Suspensions for students with 3 or more Reclaimed instructional time (not analyzed yet)

12. Referral Data SY06

13. Service Data SY07

14. MPS Student Demographics SY06-SY09 • 1017 Students • 61% male and 39% female • General Ed: 55% and 45% Special Ed (SPED 15.9%) • Ethnicity • NATIVE AMERICAN – 8.3% (4.5%) • AFRICAN AMERICAN – 48.9% (39.6%) • ASIAN – 2.1% (9.0%) • HISPANIC – 28.1% (17.1%) • WHITE – 12.6% (29.8%) • NATIVE HAWAIIAN PAC ISL – 0% • ELL – 31.2 % (23.2%) District wide percentages are in parenthesis

15. Evaluation of MPS ESMHP Access and sustained engagement in treatment 85 % of students seen once face to face; 70% within 10 days; 65% 1st time receiving services Average 17 visits per school year Improved mental health functioning Parents and teachers report decreases in the emotional and behavioral problems Improved school functioning Decrease in school suspensions for students receiving mental health treatment Principals and school social workers reported reduced office referrals and student suspensions

16. Impact on Suspensions • SY07 N=298 Seen 4x and had 1 suspension SY06 (n=82); only 27.5% had suspension in SY06 • Impact on suspensions: SY06 to SY07 – N=82 • 50% reduced 1-6 • 32% stayed the same

17. Multiply Suspended SY06-07 • SY06-SY07 Multiply Suspended (N=19) • SPED=64% • ELL=37% • Male=75% • African American=63% • 2 suspensions or less=78.9% • 52.6% to 1 or less • Moved to Zero=37%

18. Multiply Suspended SY06-SY07 (N=19)

19. Change in Mean Suspensions: SY06-SY07

20. MPS – SY07 Change Mean Suspension: Full/African American

21. Change in Attendance: SY06 to SY07 (N=35, N=20) Total Sample of Treatment kids = 159 40 (25%) had less than 90% Attendance in SY06 (5 kids missing data in SY07; sample 35 students) Total Sample of 133 comparison 20 (15%) of them had lower than 90% attendance in SY06

22. Psychosocial, Placement and Cost Outcomes in Maryland’s SMH Programs for Youth in Special Education Sharon H. Stephan, Ph.D. University of Maryland School of Medicine Center for School Mental Health

23. Acknowledgements • Maryland State Department of Education • Dr. Carol Ann Heath • Dr. Jodi King • Prince George’s County Public Schools • Dr. Pamela Downing-Hosten • Dr. Florence Foreman • Baltimore City Public Schools • Ms. Kim Lewis • Center for School Mental Health • Drs. Dana Cunningham, Eric Slade, Mark Weist, Nancy Lever • Christianna Andrews, Kerri Chambers, Emily Sidway

24. Background • Nationally, there has been an upward trend in students identified with emotional disturbances (ED) requiring intensive services to access educational standards. • This trend is evident in Maryland with the Maryland State Department of Education listing ED as a high growth category increasing approximately 75% over the last ten years. • In the five years prior to initiating the Prince George’s County SMH Initiative, the use of non-public placements in Maryland had increased by 20%, with some districts their use by over 30%.

25. Current Issues in Nonpublic Placements • Non-public placements cost local school districts over 40K per student • Many districts do not have mental health services that are devoted to students identified as disabled • There are multiple indicators that the use of non-public placements are racially disproportionate • The outcomes associated with these placements are inconsistent

26. Rationale for Implementation • Initiation of PGSMHI (2006) • Collaboration between MSDE, state university, and local school system • Concern over high rate of referrals to non-public schools • Concern over racial disproportionality in placement in non-public schools • High costs of non-publics • Relative lack of school based services

27. Rationale for Implementation • Initiation of BSMHI (2009) • Collaboration between MSDE, state university, and local school system • Concern over high rate of referrals to non-public schools • High costs of non-publics • Documented success of PGSMHI

28. Target Population Students in special education who are at risk of entering non-public settings due to an increase in behavioral and/or emotional problems Students in non-public settings who are prepared to return to their home school

29. Program Model • Therapists • Case managers • Psychiatric consultation • Family Support Partners (BSMHI)

30. Limited Local Capacity for Data Analysis • The data on highly restrictive placements wasnot organized to answer questions and guide interventions • Breaking this barrier: • Identified risk factors that contributed to restrictive placements - Age, Grade, Compliance Issues, Suspensions, Disability, Gender • Analyzed national, state, local and individual data sources • Targeted efforts on suspected risk factors • Collected extensive data on clients

31. Breaking Down Barriers • Financing • Total costs of Non-public and alternative programs are hard to determine • Non-public placements involve financing from two sources- the state and local level • Completed low and high estimates of cost savings for both the state and local district • Moved thinking from “GRANT” thinking to “DIVERSION” thinking.

32. Economic Evaluation • Determined costs associated with PGSMHI based on 3 years of data and 77 students • Personnel • Trainings • Equipment and supplies • Determined average cost of nonpublic placements • Education costs • Transportation costs

33. Economic Evaluation • Moved from dichotomous thinking ….stay or go, Emotionally Impaired or not, etc. • to every day counts = continuous variable success for each student • Question changed to how many days was the student maintained in the public setting? • Unit of analysis is the number of days prevented in non-public (not whether the student was placed) • Cost determined by the number of days placement was delayed or averted times cost differential per day

34. Costs Note: Data is based on a program year of 240 days

35. Cost Savings • $133 per student/day averted • For students receiving intense services and are considered at imminent risk for non-public placement • These savings are not expected for students who have emotional conditions but are not at risk for non-public placements • These savings are not expected for “usual care” mental health services provided in a school

36. Limitations of Evaluation • Did not assess indirect costs of adverse consequences for school staff and students of retaining students in public schools • No comparison sample of students in nonpublic setting • Assume that students would have been quickly placed in a nonpublic setting without intervention

37. Student GPA

38. Courses Passed

39. Student Observations

40. All Symptoms Youth Intake N = 59 Parent Intake N = 47

41. BSMHI Placement Outcomes • 93% of the students enrolled in the BSMHI were maintained in their current placement. • The number of students referred to non public placement at primary school decreased 50% from the 2008-2009 school year to the 2009-2010 school year.

42. BSMHI Cost Outcomes • Students were diverted a total of 24.3 years from more restrictive placements. With an estimated additional cost of $31,915* per student per year for a nonpublic placement, approximately $775,534 was saved the first year of this project. $43,000 Nonpublic Placement for 1 Year - $11,085 Public School Cost for 1 Year $31,915 Add’l Cost for Nonpublic Placement x 24.3 Placement Years Deferred $775,534.50 Cost Savings

43. Center for School Mental Health • Objective 1.2 Continue and strengthen ongoing CSMHefforts to conductcomprehensive analyses of established, data driven and impactful local and state SMH programs, with data-driven findings and case examples shared in yearly summary reports (12/11, 12/12, 12/13). • Objective 1.3 Enhance CSMH efforts to assist state and local program leaders to identify and measure targeted student and school-level outcomes and to incorporate data-based decision making into practice via a quarterly SMH outcomes/data management newsletter, annual webinar (5/11, 5/12, 5/13), and a regularly maintained compendium of online resources. • Objective 2.7 Narrow the research-practice divide and improve communication and knowledge sharing, and facilitate partnerships between researchers and other SMH stakeholders via the development and convening of a SMH Research-Practice Network and encouraging increased participation of researchers in the annual conference and National COP.

44. Call for Papers! • Special Issue on School-Based Mental Health Practice, Data-Driven School Mental Health Practice:  (How) Is It Possible? Advances in School Mental Health Promotion This issue will seek to better understand how data-driven and research-based educational interventions are being manifested in school mental health delivery.  For more information, please contact the Special Issue Guest Editor Michael S. Kelly at mkell17@luc.edu.  The deadline for manuscript submission is June 15, 2011.  Submissions should be sent to: mkell17@luc.edu

45. Discussion • Please unmute your phone so that you can participate in the discussion. • You also can type in questions on the iLinc website using the instant messaging.