National Evaluation of the ATTC Network: Findings and Recommendations

1. National Evaluation of the ATTC Network:Findings and Recommendations Richard Finkbiner, Ph.D. Project Director Megan Cummings, B.A. Margaret Gwaltney, M.B.A. Deputy Project Director Cori Sheedy, M.A. Roy M. Gabriel, Ph.D. Principal Investigator Jeffrey R.W. Knudsen, M.A. Presentation to ATTC Directors November 4, 2010

2. Overview of Today’s Presentation • Evaluation Purpose and Approach (review) • Evaluation Design and Data Collection (review) • Final Reporting Approach • 7 Findings from National ATTC Evaluation • 5 Recommendations

3. Some History: 2005–2010 • Planning the evaluation: 2005–2007 • Concept and CSAT intent for national evaluation of ATTC Network launched at Nov. 2005 ATTC Directors’ Meeting (Portland). • Abt and RMC staff continued to engage ATTC Directors and stakeholders in evaluation planning, logic model development, and instrument development through fall 2007. • OMB Instrument Package submitted fall 2007. • MANILA, Abt, and RMC competed for and won CSAT award to implement the evaluation, summer/fall 2007. • Implementing the evaluation: 2007–2010 • ATTC site visits, key informant interviews, Regional Advisory Board Surveys began winter 2008 and concluded spring 2009. • Customer Satisfaction and Benefit Survey began winter 2009 and concluded spring 2010. • Change in Practice Study began winter 2008 and concluded spring 2010. • Participation in every ATTC Directors’ meeting during this period.

4. CSAT Goals for Evaluation • Identify the successes of technology transfer efforts and build upon them in the future. • Share lessons learned for the enhancement of all regions’ activities. • Distinguish between region-specific and more cross-regional processes and outcomes

5. Review of Evaluation Design • 8 evaluation questions • 3 interconnected studies • Planning, Partnering and Service Delivery Study • Customer Satisfaction and Benefit Study • Change in Practice (CIP) Studies • 12 data collection activities • 3 secondary data (GPRA, existing reports, etc.) • 4 tied to CIP studies

6. Logic Model for the National Evaluation of the ATTC Network Activities Objectives Scope Outcomes Training Awareness Raising State/ Regional Program Outputs Technical Assistance Inputs Resources Funder Goals & Mandates Regional/ State Needs Meetings/ Conferences Mediating Factors Immediate Outcomes Skill Building Multi-Region Academic Programming Intermediate Outcomes Research Dissemination Change in Practice National Product Development Long-TermOutcomes Partnership Development ATTC NetworkEvaluation Mediating Factors include: geographic expanse/density of service population; culture/ diversity of target population; longevity of ATTC contractor; longevity of SSA directors; state/regional addiction treatment infrastructure; organizational readiness

7. Planning, Partnering & Service Delivery Study: Data Collection Activities (Review) • Site visits to each regional ATTC and the ATTC National Office, consisting of interviews and focus groups with ATTC staff and consultants • ATTC Regional Advisory Board Survey • Key informant telephone interviews with ATTC stakeholders within each region • Analyses of data from ATTC Event and Activity Reporting Database

8. Customer Satisfaction and Benefit Study: Data Collection Activities (Review) • Primary data collection activity: Customer Satisfaction and Benefit Survey (CSBS) • Secondary data: Integrating findings from ATTC regional site visits, key informant interviews, Regional Advisory Board Surveys, and GPRA.

9. Change in Practice Study: Data Collection Activities (Review) • Critical Action Surveys • Clinical Supervision (CS) • Motivational Interviewing (MI) • Treatment Planning MATRS (Tx PM) • Success Case Interviews • Audio Tape Reviews (MI only) • Organizational Readiness for Change Survey (Tx PM only)

10. Completed Data Collection Activities • 15 site visits—all ATTCs and National Office • 14 Regional Advisory Board Surveys (N = 198, 56% response rate) • N = 162 Key Informant Interviews (86%) • Includes 49 SSA/rep interviews (96%) • GPRA analysis • N = 102,000 participant records from 2005–2007 • N = 140,000 participant records from 2008–2010 • ATTC Event and Activity Reporting Database • N = 4,787 event records, 2008–2010

11. Completed Data Collection Activities (cont.) • N = 1,954 Customer Satisfaction and Benefit Surveys • N = 388 Critical Action Surveys (76% response rate) • N = 67 Success Case Interviews (93% of target) • N = 9 MI Audiotape Surveys (12% of target) • N = 16 Organizational Readiness for Change Surveys

12. Final Reporting Approach • Final report organized around 7 main evaluation findings (not presented by sub-study, evaluation question, or data collection method) and 5 recommendations • Building upon the interconnected nature of the data collections in each of the 3 studies • A synthesis of results that increases confidence in the convergent validity and robustness of these findings for program and policy development • 3 sub-study reports to follow (Jan. 2011) • Webinars for ATTCs on these more specific results • Presentations • Journal articles

13. Today’s Presentation • Finding 1: Richard Finkbiner, MANILA • Findings 2 and 3: Cori Sheedy, Abt • Findings 4 and 5: Roy Gabriel, RMC • Findings 6 and 7: Jeff Knudsen, RMC • Recommendations: Roy, RMC

14. Suggested Ground Rules for Questions and Discussion in Today’s Presentation • During presentation of each finding, restrict your questions to those of clarification of our data sources, interpretation, etc. • Reserve offering more reflective observations and implications until after all 7 findings have been presented. • Discussion of the 5 recommendations (during working lunch) can be more dynamic and spontaneous.

15. FINDING 1 • The vast majority of ATTC customers and stakeholders perceive the benefits they receive from ATTC services to be extremely valuable. Respondents who most often participated in ATTC services reported the greatest benefits. • High levels of satisfaction and perceived benefits of ATTC products and services • CSBS sample heterogeneous and representative of the addictions treatment workforce • Frequency of participation in a variety of ATTC events strongly related to perceived benefits • Motivation to participate in ATTC events • Participation in and satisfaction with specific types of ATTC services varies by role

16. Customer Satisfaction and Benefit Survey: Response Rates Source: CSBS (n = 1,954)

17. Participation in Different Types of ATTC Activities

18. Beyond Satisfaction: Perceived Benefits of ATTC Technology Transfer

19. Perceived Benefits by Varying “Dosage”of Participation in ATTC Events

20. Perceived Benefits by Participants’ Motivation for Attending ATTC Events Source: CSBS (n = 1,954)

21. FINDING 2 • ATTCs succeed in balancing the needs, interests, and requirements of diverse stakeholders within available resources. The variety of stakeholders creates a strategic tension, which ATTCs address by delivering services that have a broad reach and are less resource-intensive. • Leveraging funding resources • Relationship-building across the ATTC Network • Mechanisms for identifying workforce priorities and stakeholder needs and interests • Assessing and evaluating needs to strategically plan services • Partnering in service delivery • Bundling of services • Consequences of ATTCs meeting the diverse array of needs and requirements

22. Leveraging Funding Resources • The majority of ATTCs have actively leveraged SAMHSA/CSAT funding, and in some cases, this additional funding is considerable. • One ATTC has nearly doubled their SAMHSA/CSAT funding, with an operating budget of more than $1,000,000.—Site Visit • The ATTC [grant] is the springboard to receive these other funds.—ATTC Director • Additional funding includes supplemental SAMHSA/CSAT funding and funding from NIDA, States, local organizations, and private foundations. • ATTCs combine funding streams (ATTC funds, partner resources, program income, etc.) to provide services. • A little less than one third (27%) of ATTC activities had 2 or more funding sources.—E&A Database

23. Relationship-Building Across the Network • ATTCs work with and across a broad array of stakeholders, spending considerable time and effort identifying partners and building and maintaining relationships. • During interviews, stakeholders identified the ATTCs’ efforts to collaborate with them and to maintain partnerships as a key factor to their satisfaction with the ATTCs. • ATTCs also view the extensive effort they devote to building and maintaining relationships as crucial to determining the most appropriate services for the region. • The methods used to build relationships vary and depend on the partner, preexisting relationships or personal connections with that partner, the partner’s location, and the expected outcome of the partnership.

24. Mechanisms for Identifying Priorities and Assessing and Evaluating Needs • ATTC mechanisms for identifying priorities and needs: • ATTCs continually seek input from customers about regional priorities and needs via formal and informal ways. • Mechanisms typically include Regional Advisory and other boards, regularly scheduled and ad-hoc communications and meetings, and participation in external committees and boards. • Assessing and evaluating needs to strategically plan services: • Processes include holding annual ATTC staff retreats to discuss priorities and develop the ATTC’s work plan, conducting and analyzing needs assessments, working directly with the SSAs to prioritize the State’s needs, and utilizing decision-making tools. • In responding to different needs, often a single activity will try to meet the needs of several different stakeholder groups. • Notably absent in the planning of services is a discussion of the expected outcomes of the activities planned. 24

25. Partnering in Service Delivery • ATTC partnerships serve as a key platform for effective service delivery and are an effort to extend the ATTCs’ reach to more participants and to obtain additional support and funding for their activities. • ATTCs partner with regional, State, and local organizations and agencies; national and Federal organizations; and other ATTCs, among others, to deliver services. Partners bring different resources and expertise to the relationship. • 84% of events between January 1, 2008, and June 30, 2010, (4,031 events) were delivered in partnership with other organizations and 23% of events had multiple partners. Most frequently cited partners included SSAs (41% of events), another ATTC (24%), State/regional organizations (23%), and community treatment programs (10%). —E&A Database 25

26. Bundling of Services • “Bundling” of services occurs when a series of ATTC activities is coordinated at regional and State levels or when services are delivered as a series on a specific topic. • More than one third (38%) of ATTC events were delivered as part of a series and in collaboration with other services. —E&A Database • Bundling of services has been shown to facilitate successful implementation of EBPs and organizational change (Fixsen & Blase, 2009)1. • During interviews stakeholders said there is a need for the ATTC to provide services aimed at changing practice and to increase follow-up activities, such as technical assistance provided over a period of time. 1Fixsen, D.L. & Blasé, K.A. (2009, January). Implementation: The missing link between research and practice. NIRN Implementation Brief No. 1. Chapel Hill: The University of North Carolina, FPG, NIRN. 26

27. Consequences of Meeting Diverse Array of Needs and Requirements • ATTCs are successful at balancing the needs of their stakeholders and providing training and technology transfer activities to meet identified needs within limited funding. • However, ATTCs encounter challenges in their efforts, e.g., • Having to build and maintain relationships with many stakeholders • Delivering services to address diverse workforce needs • Coping and operating with limited resources to meet identified needs and other Federal requirements • The result is that a disproportionate emphasis is placed on services and activities that fall within the earlier phases of the technology transfer continuum with fewer activities aimed at changing treatment practice. 27

28. FINDING 3 • The ATTCs have adopted a State-specific approach to service delivery, built upon the strong relationships they have established with SSAs. Additional demands on this relationship will likely arise from recent national policy developments and challenging economic conditions. • Relationship-building with SSAs • Longevity, expertise, and shared values of ATTC and SSA staff • Policy and economic conditions affecting states and the addictions treatment field

29. Relationship-Building With SSAs • ATTC directors and staff invest significant time and employ multiple strategies to initiate and nurture their relationships with SSAs. These strategies include: • Including SSA directors on ATTC Regional Advisory Boards. • Nearly three fourths (73.3%) of SSAs or their representatives agree or strongly agree that their input is sufficiently considered in this process, virtually the same as that of all Regional Advisory Board members (73.4%)—Regional Advisory Board Survey • Scheduling regular calls/meetings between ATTC staff and SSA directors. • Participating on outside committees or organizations, where SSA directors, staff, and other leaders in the addictions field also are represented (e.g., State provider associations or SSA committees). • Having ad hoc meetings with SSA directors at conferences and other regional or national meetings. 29

30. Longevity, Expertise, and Shared Values of ATTC and SSA Staff • ATTCs have unique expertise that is an asset to the SSAs, and this expertise has helped build, solidify, and sustain the relationship between SSA directors and ATTCs. • A core function of the SSAs is providing training and assistance to professionals in the field, and SSAs are therefore motivated to form close relationships with their Regional ATTCs. • Although SSA Administrators interviewed have only been in their positions an average of 3.5 years, they and ATTC Directors have similar longevity in the field (ATTC Directors: average of 26.5 years; SSAs: average of 22.3 years), which facilitates establishing and maintaining relationships. 30

31. Policy and Economic Conditions Affecting States and the Addictions Treatment Field • Contextual factors also affect the relationship between SSAs and ATTCs and suggest their intensity and importance may increase in the near future. These include: • Current economic recession, budget constraints, furloughs, travel freezes, program cuts. • Partnering with the ATTC is even more important in leaner times when the State has fewer resources. We are such a small State and have a declining budget, so that partnering with the ATTC has been invaluable. We wouldn’t have been able to do training on returning veterans or MI without the ATTC.—SSA Director • Changing profile of individuals with substance use disorders and environment in which individuals are being treated. • Continuing high turnover in frontline staff and supervisors. • Current health care reform and new requirements for parity, which will lead to increasing numbers of insured individuals eligible for health coverage and behavioral health treatment. 31

32. FINDING 4 • Many stakeholders see a need for the ATTCs to establish a national identity with regard to strengthening the workforce within the national behavioral health infrastructure. • Shifting federal direction and mandates to the ATTC Network • Increased collaboration among the Regional ATTCs • Dissemination and coordination by the ATTC National Office • ATTCs moving toward a leading role in the national behavioral health system

33. Some History . . .  • ATTCs initially funded in 1993 • N = 11 “independent Addiction Training Centers” • U.S. not fully covered and no National Office • Expanded in 1998 • N = 13 regional ATTCs and a National Office (no clear role) • Collaboration: The Change Book, TAP 21 • Topically-focused inter-regional committees • A few years later • ATTCs told to move away from national collaboration and stick to meeting regional needs • GPRA “targets” • NIH/SAMHSA role demarcation

34. More Recently . . .  • Increased collaboration • 2007 RFA and direction from GPO • Restoration of inter-regional work groups • Reduce duplication and capitalize on regional “lessons learned” • TAP 21-A, ATTC Technology Transfer Model, ROSC • Expanded, proactive role of National Office • Consolidated ATTC Website, Event and Activity Reporting Database, Leadership Institutes, Workforce Survey, use of technology, etc. • National point of contact w/other professional groups (e.g., NADAAC, NASADAD) • Funding opportunities for the Network

35. The Changing Landscape Behavioral Health is an Essential Part of Health —SAMHSA • ATTCs’ proven proficiency in establishing partnerships across wide variety of constituencies. • Data from this evaluation documenting benefits of ATTC work (beyond satisfaction) from a variety of perspectives . • Establishing partnerships with mental health and primary medical care professionals and organizations: • National Association of State Mental Health Program Directors • National Council for Community Behavioral Health Care • Federally Qualified Health Centers, Community Health Centers • A consistent presence and voice in SAMHSA internal and inter-agency work groups. • Providing skill-based technology transfer to medical professionals (e.g., SBIRT). • Continuing to educate other professionals, disciplines.

36. Information Source • Not a universal feeling, but among those adding this to their “wish list” for ATTCs are stakeholders from: • CSAT • NASADAD • Research Society on Alcoholism • NADAAC • Advocates for Human Potential • SSAs • Addiction Educators • State Providers’ Associations

37. FINDING 5 • Following their participation in ATTC initiatives designed to change clinical practice, practitioners report high levels of implementation of evidence based practices (EBPs) in addiction treatment settings. Implementation efforts were accompanied by improvements in associated clinical skills and improvement in supervisor/clinician/client relationships. • High levels of implementation of critical actions • Improved implementation and EBP-related skills • Key ingredients to successful implementation • Improved clinical relationships • Assurance of fidelity of implementation requires more in-depth study

38. Methodology and Data Sources (Review) “Success Case” method • Critical Action Surveys • N = 100 MI participants (70% response rate1) • N = 150 CS participants (79%) • N = 138 Tx Planning MATRS (79%) • Success Case Interviews • N = 24 MI (100% of target sample size) • N = 21 CS (88%) • N = 22 Tx Planning MATRS (92%) 1 Of eligible and consenting participants

39. “Critical Actions” of EBPs • To assess changes in practice, needed to get specific—How would we know that a clinician was putting into practice what they learned via ATTC technology transfer activities? • Through review of research literature and ATTC curricula, identified “critical actions” associated with each of the 3 topics • 5 for MI (e.g., Seeking to understand your client’s frame of reference via reflective listening) • 6 for CS (e.g., Conducting regular supervisory interviews with supervisees) • 7 for Tx PM (e.g., Producing individualized treatment plans that include goals reflecting what the client wants to achieve) • These critical actions were the centerpiece of our Critical Action Surveys (“Since participating in the ATTC . . . to what extent have you been able to implement the following . . . ?) • Formed a composite of the 5, 6, or 7 critical action implementation responses for a “total score” of implementation of a given EBP

40. High Prevalence of Implementation of Critical Actions after ATTC Technology Transfer Event(s)

41. Other Improvements Reported after ATTC Technology Transfer Event(s) • Significant improvement in proficiency with these critical actions (all 18 at p < .001) • Improvements in related clinical skills or agency operations • MI: 5 skills, 97%–99% reporting improvement • CS: 4 skills, 91%–98% reporting improvement • Tx PM: 4 skills, 61%–83% reporting improvement • Perception of improved client behaviors such as client engagement, retention in treatment • MI: 62%–83% • Tx PM: 50%–60% • Reported by many interviewees: Improved clinical relationships • MI and Tx PM: Clinician and Client • CS: Clinical Supervisor and Clinician

42. Key Ingredients to Successful Implementation • Individual Clinician characteristics • Belief that the practice was important to quality care • Prior experience/familiarity with the critical actions • Treatment Agency/Organizational characteristics • Formal support from clinical supervisor(s); informal support and recognition from management • Work environment supportive of client-centered approaches to care • ATTC TT Event(s) characteristics • Expert trainer, highly interactive format, peer-to-peer exchange • Evidence-based content, concrete, user-friendly materials (checklists, charts, forms)

43. FINDING 6 • While the ATTCs deliver high quality services to meet the needs of diverse stakeholders and customers, there is less emphasis on matching participants to services and providing posttraining support to improve technology transfer outcomes. • ATTC services match needs and requirements of stakeholders and are highly rated • Recruiting participants to match the services provided and outcomes desired • Postevent support enhances technology transfer outcomes

44. Service Delivery Continuum • Individual, organizational, and event characteristics all play a role in supporting implementation. • In considering these data in the context of a full service delivery continuum, multiple lessons emerge.

45. Clearly Identified Strengths of ATTC Service Delivery • ATTCs successfully interact with a broad array of customers and stakeholders, resulting in a series of service delivery topics that are consistent with the needs and interests of ATTC stakeholders and customers and the requirements of their funders. (c.f., Finding 2) • Customers report a high level of benefit and satisfaction from the activities, events, and products provided by the ATTCs. (c.f., Finding 1) • Customers described specific characteristics of ATTC events that aided their learning and eventual implementation of critical actions associated with EBPs (c.f., Finding 5):

46. Importance of Recruitment in Achieving Technology Transfer Outcomes • Critical Action Surveys and subsequent success case interviews offer compelling evidence that participants’ success in postevent implementation is based more on the right match of service and participant than the length or intensity of the event(s). • A participant’s prior experience with the practices and postevent support provided were the strongest determinants of postevent implementation success (regardless of self-reported proficiency). • Many ATTC participants reported not being in an appropriate role to implement various critical actions, jeopardizing many potential technology transfer outcomes. • Differential recruitment processes have varying levels of control over who attends ATTC events—hugely important especially for events tagged with “change in practice” objective. • ATTC-controlled versus shared or externally controlled recruitment strategies • Targeted versus open recruitment

47. Importance of Post Service Follow-Up & Support in Achieving Technology Transfer Outcomes • Data clearly indicate that participants need and want both formal and informal posttraining support to have meaningful, lasting implementation success. • Specifically, they need further support to assist with (a) making the new practice a priority, (b) getting better at the practice they are implementing, and (c) avoiding the tendency to “fall back into their old ways” of doing things. • Participants cited multiple organizational- and ATTC-based supports that aided their successful implementation.

48. FINDING 7 • ATTC services produce improvements in the knowledge, skills, and practices of the addictions treatment workforce. However, limited resources appear to be devoted to the specification and measurement of the intended outcomes of these services. • ATTCs’ technology transfer efforts bring about a variety of positive changes • Defining technology transfer success remains a challenge • The ATTC Network technology transfer model: boundaries and definitions • Lack of clarity in intended outcomes of ATTC technology transfer activities • The benefits of improved clarity in the intent and goals of ATTC technology transfer

49. The Work of ATTCs Results in Positive Changes in Their Participants • As reported previously (c.f., Findings 1 and 5), multiple data sources in this evaluation indicate that ATTC service participants report positive changes in their knowledge, skills, and practices following ATTC events. Specifically: • Statistical analysis of pretest, posttest, and 30-day follow up GPRA over the first 3 years of the current funding period demonstrates participants perceive a statistically significant increase in their knowledge, skills, and effectiveness from pre- to postevent and from post- to 30-day follow-up of the event (both p < .01). • This perceived improvement is also supported by findings from the CSBS, with 95% of those respondents expressing agreement that they increased their knowledge, 91% that they increased their skills, and 90% that they were able to do their job better as a result of services received from the ATTCs. (c.f., Finding 1). • Critical Action Surveys and follow-up interview results further confirm the ATTCs’ impact on clinical practice as training participants reported high levels of implementation, increased proficiency, and significant impacts on their clinical practice (c.f., Finding 5).

50. Defining Technology Transfer Success is a Challenge • Need for common terms and definitions: • Several models or frameworks for technology transfer • Corresponding variety and inconsistency in their terminology related to important concepts such as “success” • Although the literature leaves little doubt that the ultimate goal when dealing with evidence-based practices is full implementation with fidelity and/or rational adaptation, smaller improvements to practice can be worthwhile. • Changes may occur in multiple, incremental stages of implementation, distinguishing between installation, initial implementation, and full implementation (Fixsen et al., 20051) of practices. • While it is clear that ATTC technology transfer efforts can elicit incremental changes and the adoption of discrete behaviors associated with EBPs, it is difficult to assess their success without knowing their ultimate goal or intention of service delivery. 1Fixsen, D., Naoom, S.F., Blasé, K.A., Friedman, R.M. & Wallace, F. (2005)Implementation research: A synthesis of the literature. FMHI Publication No. 231 Univ. of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network