Quality Monitoring in New York State: ACT Licensing, Client Outcomes, and DIGS Quality Indicators

1. Quality Monitoring in New York State: ACT Licensing, Client Outcomes, and DIGS Quality Indicators Anthony Mancini, PhD Bureau of Adult Services Evaluation Research New York State Office of Mental Health

2. Overview • NYS ACT Licensing Initiative • Development of Fidelity-based Licensing Protocol for ACT • Outcomes Monitoring (CAIRS) • Evaluation of ACT Licensing • Focus on DIGS: Deriving Quality Indicators for Adult EBPs from Administrative Data

3. 1. NYS ACT Licensing Initiative • Background • Rationale • Challenges

4. Background for ACT Licensing • Part of the state’s “Winds of Change” campaign to increase access to evidence-based practices (EBPs) • Switch to Medicaid billing • Expand ACT teams from 27 to 60 in first phase • Make fidelity to ACT & EBPs linchpin of implementation • Make ACT a platform for all EBPS • Training in ACT model and all EBPs

5. Rationale for Fidelity-Based Licensing • Promote a well-validated treatment for high-need clients • Through licensing, can promote quality of care, i.e., high fidelity • Fidelity provides index of success for MHA • Clarify requirements for participating programs • Allows for focused technical assistance • Identifies additional training needs

6. Challenges for ACT Licensing Initiative • Can licensing and certification staff reliably use the DACTS? • Is the DACTS too time consuming to administer in the context of a licensing visit? • Will licensing staff accept a new methodology for evaluating programs? • How do we integrate assessment of fidelity and traditional state standards? • Should fidelity items be adapted to match NYS standards, or is it better to use the DACTS as is to allow for comparison with national data?

7. 2. Development of a Fidelity-Based Licensing Protocol • Process • Outcome • Example

8. Development Process for ACT Licensing Protocol • Working group of various stakeholders • Lengthy iterative process taking over 1 year • Developed content, language, methodology, anchors for each item • Reviewed by senior OMH staff, ACT providers, and county representatives • Trained licensing and certification staff • Revisions based on two years of use • Cut points for scoring

9. Outcome: ACT Licensing Instrument • 56-item survey (example to follow) • Modeled on Dartmouth ACT scale; that is, uses 5-point behaviorally-anchored items • Integrates into one survey traditional outpatient and fidelity standards • Includes standards that address recovery-oriented practice, an increasing focus of concern

10. Outcome: ACT Licensing Instrument (cont’d) Domains included: • DACTS (excluding 2 items; α 0.68) • NYS Staffing requirements (e.g., % of professional staff) • EBP fidelity (e.g., family psychoed., wellness management; α 0.41) • Recipients’ rights (e.g., cultural comp., grievance procedure, recipients’ rights, recipient choice; α 0.78) • Case records (e.g., assessment & treatment planning; α0.83) • Program operations (e.g., policies & procedures related to incident review, quality improvement; α0.81 ) • Program site (e.g., furniture, appropriate storage of medications & case records; α0.64)

11. Example of Fidelity-Based Licensing Indicator

12. 3. Outcomes Reporting & Monitoring (CAIRS) • Background • Features • Domains • Reports

13. Background on Outcomes Reporting System (CAIRS) • Child and Adult Integrated Reporting System (CAIRS) • System evolved out of need to track child services • Need to standardize reporting across different program types • Now collects data for Child Services, ACT, AOT, Case Management, and Housing Programs • Will be extended to other programs, including Personalized Recovery-Oriented Services (PROS)

14. Features of CAIRS • Secure and confidential • HIPAA compliant • Web-based • Collects client outcomes data from program staff • Data submission required at baseline and every 6 months thereafter for every client

15. Domains Reported on in CAIRS • CAIRS collects data on a variety of outcomes, including: • Psychosocial functioning • Housing status (e.g., type of residence) • Employment (e.g., current and past, competitive vs. sheltered) • Educational achievement (and current activity) • Quality of relationships with others • Self-care (e.g., ADLs) • High risk (e.g., violence toward self or others) • Substance abuse (e.g., type and level of use) • Medication regimen, side effects, and adherence

16. Domains (cont’d) • Type of services in treatment plan • Medical diagnoses • Entitlements and income sources • Relationship status, child custody status • Inpatient psychiatric utilization history (up to past 24 months) • Criminal justice involvement • Advanced directive or health care proxy

17. Reports Generated by CAIRS • Reports generated via web-based application • Aggregated at different levels (e.g., consumer, program, county, region, state) • Flexibility in report generation • Instant feedback to clinicians and administrators on individual clients and across clients

18. 4. Evaluation of ACT Licensing • Fidelity • Clients • Outcomes

19. 4a. Fidelity • Questions: • What is the impact of licensing on fidelity? • Where do programs excel and where do they have difficulty? • What correlates with higher fidelity?

20. Fidelity ACT fidelity was very good overall: high mean fidelity scores and higher in state PCs

21. Fidelity Key ACT fidelity items showed high levels of implementation

22. Fidelity However, teams struggled in areas related to service intensity and specific services

23. Fidelity A number of program leader characteristic predict ACT fidelity

24. Fidelity Fidelity scores were highly correlated with other licensing domains

25. 4b. Client Characteristics & Appropriateness • Questions: • Who gets ACT in NY? • Are appropriate clients being enrolled?

26. Who Gets ACT in NY? Demographics & Diagnosis

27. Are ACT Clients Appropriate?

28. 4c. Client Outcomes • Questions: • Does enrollment in ACT reduce inpatient and ER utilization? • Does enrollment in ACT reduce inpatient and ER cost?

29. Client OutcomesMCD Inpatient/ER Utilization & Costs for New Admissions (N=438)

30. Client OutcomesMCD Inpatient/ER Utilization & Costs for New Admissions (N=438)

31. 5. Focus on DIGS: Deriving Adult EBP Quality Indicators from Administrative Data • Context • Goals • Data • Conceptualization • Indicators • Next steps

32. New York State Context for DIGS • New York’s Winds of Change initiative to support EBPs • Roll-out and licensure of ACT teams with focus on EBP fidelity • Data systems for assessing outcomes, service delivery, and treatment planning (CAIRS) • Personalized Recovery-Oriented Services (PROS) programs

33. Goals for DIGS Overarching goal: To meet reporting requirements for Federal Data Infrastructure Grants (focus on Tables 16 & 17) • To achieve that goal, we have: • Identified potential data sources • Assessed their relevance to EBP reporting • Conceptualized reasonable standards for EBP conformance

34. Data Sources for DIGS • Licensing data on ACT • Program level • Fidelity to ACT, IDDT, SE • EBP Implementation data (toolkit, FPE projects) • Program level • Fidelity to IMR and FPE • CAIRS Data (ACT, case management, housing providers) • Client level • Services and treatment planning components • Medicaid Data (all billable services) • Client level • Services • NYISERS (vocational programs) • Client Level • Services

35. Conceptualizing Performance Indicators for DIGS • Question is: How to operationalize EBP quality indicators using available data? • Focus on: • Supported Housing • Supported Employment • Famly Psychoeducation • ACT • Integrated Treatment for Dual Disorders • Illness Management and Recovery

36. Conceptualizing Performance Indicators for DIGS (cont’d) • Developed algorithms/methods for each of the EBPs using different data sources • Census of programs (CAIRS) • Services provided (CAIRS; Medicaid) • Process-based data on fidelity or quality of service (ACT licensing data; EBP projects data)

37. Proposed DIGS Performance Indicators(Table 16) • Supported Housing (SH) • Method Using CAIRS (Penetration only) • All persons reported in CAIRS to be receiving SH • Relatively straightforward because no fidelity measure needed

38. Proposed DIGS Performance Indicators(Table 16 cont’d) 2. Supported Employment (SE) • Method for ACT Teams Using ACT Licensing and CAIRS Data • If teams meet these fidelity conditions: • Team has employment specialist (score = 5) & • Employment specialist provides majority of service (score = 4 or 5; e.g., job development, coaching, follow-along support) & • There is some diversity of job options (score = 3-5; e.g., not all jobs are working as messenger for agency) • Then # of clients with SE in treatment plan on that team (CAIRS) represents penetration and reporting for that indicator

39. Proposed DIGS Performance Indicators(Table 16 cont’d) 2. Supported Employment (cont’d) • Method Using Medicaid Data • # of persons with Vocational Services rate code in Medicaid (Penetration) • Rate code means that person has 15 hours/week in setting that integrates mental health and vocational services (Proxy for fidelity) • Method Using NYISERS Data for Vocational Programs • # of persons receiving SE • No fidelity data as yet

40. Proposed DIGS Performance Indicators(Table 17 cont’d) 3. Assertive Community Treatment • Method Using CAIRS Data • # of persons receiving ACT services (penetration) • Because structural features of fidelity, e.g., staffing, meeting structure, in place for all teams in NY, no decision rule 4. Family Psychoeducation (FPE) • Method Using Data from FPE implementation initiative • Collaboration with University of Rochester • 21 programs running family groups • About 100 clients; more as project continues • Fidelity monitoring by phone • Data are still being collected • No standards or algorithm as yet

41. Proposed DIGS Performance Indicators(Table 17 cont’d) 5. Integrated Dual Disorders Treatment (IDDT) • Method for ACT Teams Using Licensing and CAIRS data • If teams meet these fidelity conditions: • Has a substance abuse specialist (score = 5) & • Provides some individual substance abuse treatment (score = 3-5) & • Provides some group substance abuse treatment (score = 4 or 5) • Then the clients on those teams with substance abuse as part of their treatment plan (CAIRS) would be counted toward penetration and reporting for that indicator on ACT

42. Proposed DIGS Performance Indicators(Table 17 cont’d) 6. Illness Management and Recovery (IMR) • Method Using Data from Pilot Project: • 4 programs • Approximately 80 clients in total • Fidelity data collected every 6 months for 2 years • No algorithm or standards as yet 7. Medication Management (MM) • No fidelity data available as yet • Refinement of fidelity scale ongoing in national project OMH is participating in

43. Next Steps • Further refinement of licensing indicators for ACT • Revision of licensing instrument • Cutpoints to reflect reasonable expectations • Ongoing evaluations of peer perspectives, psychosocial outcomes, and recovery-oriented practices • CAIRS outcomes application • Extension to Personalized Recovery-Oriented Services (PROS) • Client-completed form rather than clinician-completed

44. Next Steps (cont’d) • Licensing for PROS • Further integration of fidelity and licensing standards • Development of indicators for recovery-oriented practices • DIGS • Ongoing refinement of data collection and analytic approach for EBPs and quality monitoring generally • Continuous quality improvement approach reflected in all initiatives