In it for the Long Haul: The Developmental Stages and Challenges of Systems Change

1. In it for the Long Haul: The Developmental Stages and Challenges of Systems Change Doug Easterling, Ph.D. dveaster@wfubmc.edu Liz Arnold, Ph.D. earnold@wfubmc.edu Reclaiming Futures Leadership Institute Miami, FL May 18, 2011 (revised)

2. Purpose of the Session • Present a “developmental model” that describes how systems-change initiatives play out over the long run • Intended to serve as a road map that can be helpful to those involved in a systems-change initiative (e.g., RF) • What challenges can you expect given your initiative’s stage of development? • Use data from the evaluation of the North Carolina RF initiative to make the model more concrete 3. Group discussion and collective problem-solving on how to navigate the key challenges associated with each stage 4. Critique the model • Are these the right stages? • Are these the key challenges? • What’s missing?

3. Format of the Session • Overview (Doug) • Systems-Change Initiatives • The Developmental Model • Presentation and Discussion around each stage of the model • Description of the stage and key challenges (Doug) • Data and impressions from the NC evaluation (Liz) • Initial thoughts from someone who has lived the process (Benjamin) • Group discussion • How has this stage played out across different RF sites? • Which challenges have been most significant and/or flummoxing? • Promising approaches to addressing key challenges • Final Discussion

4. WFU Evaluation of North Carolina RF Initiative • Evaluation Question: What happens when the RF model and the associated supports (e.g., funding, fellowships, technical assistance, coaching) are introduced into six NC communities? • Formation and functioning of local RF teams • Changes in procedures used for screening, assessment, referral, etc. • Introduction of new tools for assessment and screening • Improved monitoring and tracking • New services and programs • Improved coordination • New partnerships • Do more youth receive appropriate services? • Evaluation Methods • Annual interviews (telephone or in-person) with individuals in fellowship positions • Periodic web-based survey completed by an expanded set of players with knowledge of the local system

5. Intent of Systems-Change Initiatives • A systems-change initiative goes beyond first-order “solutions” to generate change that is both system-wide and systemic • Not about introducing or changing a specific program … • … but rather a change in the “system” through which services are delivered and/or problems are solved • Exert leverage on structural factors that influence bottom-line outcomes • Beyond superficial fixes to root issues

6. What’s a “System”? • System = multiple players, processes, and revenue streams that are organized to accomplish a larger purpose • Where do “systems” operate? • A large institution (e.g., courts, medical center) • A “cluster” of organizations that are focused on a common issue or population (e.g., mental health treatment providers) • Across an entire community • acknowledges the breadth of actors who have influence over any complex issue • What sorts of changes allow systems to become more effective? • New programs and services that address gaps in the system • Improved programs and services • Better tracking and monitoring of outcomes – with feedback loops to improve services • Improved coordination among actors (e.g., referrals, common intake, shared or linked data systems, collective problem-solving) • Culture change that promotes excellence, learning, and ongoing improvement • Any other change that allows “clients” (patients, customers, families) to meet their needs and achieve their goals

7. Systems-Change Initiatives: Approach • Inputs and Tools for Systems Change • Inter-departmental, inter-agency, or community-wide stakeholder group • Front-end assessment • Needs assessment • Environmental scanning • Evaluation of current programs and services • Strategic planning and thinking • Vision of what the system should look like • Identification of strategic goals and leverage points • Theoretical Framework with pathways to progress • Consultants and Technical Assistance • Opportunities for Learning • A systems-change initiative can originate from either: • Local actors who recognize the need for change, or • An outside agent looking to make change happen in multiple places • Private foundations • Government agencies • National or regional organizing groups (e.g., IAF,People’s Institute)

8. How is RF a systems-change initiative? • Change involves new services, programs and approaches throughout an entire system, not just within specific organizations • Pushes for changes in how (and how well) organizations work with one another • Multi-agency coalition provides leadership, carries out planning, develops and implements strategy, and monitors progress • Local strategies are informed by a theoretical framework • Technical assistance and opportunities for learning (from NPO) • Accountability comes through tangible improvements in youth outcomes (rather than simply doing new things)

9. As a funder (Colorado Trust) Colorado Healthy Communities Initiative Teen Pregnancy Prevention Initiative Violence Prevention Initiative Colorado School Health Education Initiative Assets for Colorado Youth (Search Institute) As an evaluator Reclaiming Futures (Kate B. Reynolds) Free to Grow (RWJF) Strategic Approaches to Community Safety (SACSI) [NIJ] Community Foundation of Greater South Wood Co, WI Duluth-Superior Area Community Foundation As a design consultant to foundations that do systems change Mary Reynolds Babcock Foundation Winston-Salem Foundation Community Foundation of Greater Greensboro National Social Capital Learning Circle (19 foundations) As a member of a systems-change coalition Clinical Translational Science Award (NIH) Out-of-school education for youth (Wallace) As a facilitator ECHO Council (social capital coalition in Winston-Salem) Systems-Change Initiatives I have known

10. Messy Difficult Ambitious Complex Rife with conflict Political Can threaten those in charge (even if they started out as an advocate or ally) Can lead to burn out Non-linear Ups and downs Progress can emerge out of apparent failure Some successes are short-lived The work is never done Different issues arise at different points What can we say about systems-change work?

11. The Key Challenge:Moving from Collective Conversation toCollective Impact (Kania & Kramer, SSIR, 2011)

15. Stages of Development • Systems change initiatives have a natural life cycle, defined by predictable stages of development • Moving to the next stage allows a greater level of impact • … but it also raises new challenges that threaten the survival of the initiative

16. Stages in the Life Cycle of an Externally-funded Systems-change Initiative

17. Timing of the Stages (typical)

18. Stage 2 Stage 3 Stage 4 Stage 5

19. What actually happens in each stage? • An overall task that sets the stage for the work • A series of more concrete tasks • Predictable challenges • Coping with those challenges • (hopefully successful coping, at least for the most part)

20. Stage 1: Proposal Writing • Defining Task: Win one of the grants • Key Tasks and Challenges • Assemble the required partners • Become versed in the initiative’s change model • Determine how the model could work locally • Write a competitive proposal

21. Stage 2: Team Development • Defining Task: Establish an effective infrastructure to carry out systems-change work • Key Tasks and Challenges • Reassemble the required partners, this time to do the real, long-term work • Learn from the funder, NPO, TA provider, evaluator, etc. what is actually expected • Work through confusion, disagreements, competing interests, personality clashes, etc. • Agree on (or at least decide on) approach, structure, goals, processes

22. Themes from the NC RF Evaluation re: Stage 2 • All 6 sites built functioning RF teams consisting of Project Director, four RF Fellows, and other key players • Teams generally were meeting 1 or 2 times per month • Some turnover in teams occurred once the grant was awarded and the work began in earnest • Most of the turnover was job-related • Some of the roles that are key to RF (e.g., treatment, community fellows) tend to be inherently transient • Notable move up by Robin Jenkins to leadership role within NC DJJDP • Conversely, some fellows and partners kept their jobs, but fell off from RF • Different level and type of contribution for pre-award vs. post-award • Project directors were pretty stable for first year • First of 6 PDs to leave left approx 1 year into RF (took a new job and left the area) • Teams began to come together and gain cohesion • Easier and smoother in some sites than others • Overall, 86% of Round 2 interviewees reported that their team worked together “great” or “pretty good” • But see chart on next slide

23. Team Functioning • “How well does your team work together?” • Site A: 4-“great” 1-“pretty good” • Site B: 4-“great” 1-”pretty good” • Site C: 2-”great” 2-”pretty good” • Site D: 1-”great” 4-”pretty good” • Site E: 1-”great” 1-”pretty good” 2-”mixed” • Site F: 3-”pretty good” 2-”mixed” • Most sites report improvement over time in communication and understanding one another’s perspectives

24. Stage 3: Initial Impact • Defining Task: Accomplish a concrete, meaningful act of systems change • Key Tasks and Challenges: • Carry out a strategic analysis of what needs to change within the system (in line with the initiative model) • Identify 1-3 specific issues that allow strategic leverage • Develop a plan for making that change (with specific responsibilities assigned to willing partners) • Carry out the plan as well as possible • Work out who does what and who is really in charge • Deal constructively with transition, resistance and setbacks • Evaluate, learn, and adjust

25. Themes from the NC RF Evaluation • In line with RF model, the 6 teams explored what might be done to improve screening, assessment, and treatment within their community • Focused initially on screening and assessment • Looked at ways to increase the number of youth screened and assessed, as well as whether different tools might be needed • Strong encouragement from NPO and DMH to adopt the suite of GAIN tools

26. Changes in Screening • Prior to RF: • the standard approach to screening was for Court Counselors to administer DJJDP Risk & Need Assessment • Other tools in limited use: GAIN-SS, MASSI, MAJORS screener • Screening was NOT carried out on all youth entering the court system • 1 ½ years into RF: • All six sites had implemented more systematic approaches to screening (including conducting screening in more locations) • In all sites, interviewees report an increase in # youth screened • In most sites, interviewees report that Court Counselors are explicitly seeking to screen all youth entering court system • Standard screening tool in all sites: GAIN-SS • In 5 sites, an interviewee mentioned that RF team had developed specific criteria for referring youth to assessment

27. Changes in Assessment • By the time of the second interview, there had been some improvements in the assessment process, but more modest than what occurred with screening • In 3 sites, interviewees report improvement in scheduling assessments, or reduced time between screening and assessment • Limited reports of more youth being assessed • Some critical review of tools that are used for assessment • In 2 sites, interviewees report change in the tools that providers are using for assessment (GAIN-I, JASAE) • Use of GAIN tools for assessment: • In no site is a GAIN assessment tool being used universally by MH/SA Tx providers • In 1 site, a key provider is using GAIN-I • In 2 other sites, some providers have been trained in GAIN-I or GAIN-Q • In remaining 3 sites, there appears to be little to no interest in using GAIN assessment tools

28. Stage 4: Extending the Impact • Defining Task: Move from a narrow win to accomplishing broader scale changes in the system • Key Tasks and Challenges: • Revisit the strategic analysis, this time with benefit of real-world experience and additional data • Decide what to do next • new approach to an unsolved problem, • address a new aspect of the system, and/or • reach a new population • Identify and recruit additional partners who are crucial to success • Revise and extend the initial strategic plan to be more comprehensive, focused and realistic • Carry out the plan as well as possible • Deal constructively with transition, resistance and setbacks • Evaluate, learn, and adjust

29. Themes from the NC RF Evaluation • Success with improved screening provides a concrete point of departure for further changes to the system • Different sites focus on different areas for change • Assessment • Changes that ensure more youth are assessed • Do assessment sooner after screening • Not a lot of emphasis on changing the tools for assessment • Treatment • In some sites, key services do not exist or slots are limited • Special problems for: • youth without Medicaid • youth who don’t speak English • Limited progress in adding new services and programs • Mentoring • Continue to try to develop a data system to track RF outcomes • Initial work on Community Engagement

30. Stage 4 Challenges and Struggles in NC • How to maintain progress (or even traction) with all the funding cuts for services and changes in mental health system • Shortage of funding for any services that go “beyond treatment” • Many struggles with collecting data from different providers and combining data into a common database. • Almost everything is done “by hand” in excel spreadsheets • Looking for ways to tie into NC DJJDP data system (NC-JOIN) • Experimenting with new templates and overlays • Teams get frustrated as the work gets more diffuse and complex • During Round 2 interviews, 86% reported that their team worked together “great” or “pretty good” • Down to 64% during Round 3 interviews

31. Stage 5: Sustaining the process of innovation and reform • Defining Task: Become a permanent venue and force for stimulating whatever changes in the system need to occur as times and conditions change • Key Tasks and Challenges • Assess and acknowledge accomplishments to date • Recognize that the coalition and its work are “part of the system” (no longer a “special” effort to change the system) • Revisit mission: Identify what needs to happen for innovation and reform to continue (rather than protecting the initial round of systems change) • Evolve the structure to meet the new mission. Where should the work reside? • Find the resources to support the new work • Enter into a new phase of innovating, planning, initiating change, assessing, learning, and evolving.

32. Themes from NC • The 6 NC sites are just beginning to think about this stage (i.e., what RF looks like when it is institutionalized) • The prospect of the KBR grant ending has stimulated questions about “long-term RF” • What position might the PD convert into? • Which funded positions overlap with RF’s intent? • Where should RF staff reside over the long haul? • Opening of a state RF office in NC brings up parallel questions re: institutionalizing RF throughout the state.

33. Final Discussion