Introduction & Training | City, ST | November 1-2, 20109

1. Local Site Children’s Behavioral Health Initiative Orientation to the Team Observation Measure (TOM) Introduction & Training | City, ST | November 1-2, 20109 Eric J. Bruns, Ph.D. & April Sather, MPH University of Washington School of Medicine Wraparound Evaluation and Research Team National Wraparound Initiative www.wrapinfo.org

2. What is Wraparound? • Wraparound is a family-driven, team-based process for planning and implementing services and supports. • Through the wraparound process, teams create plans that are geared toward meeting the unique and holistic needs of children and youth with complex needs and their families. • The wraparound team members (e.g., the identified youth, his or her parents/caregivers, other family members and community members, mental health professionals, educators, and others) meet regularly to implement and monitor the plan to ensure its success.

3. Theory of change for wraparound process Ten principles of the wraparound process • Intermediate outcomes: • Participation in services • Services that “work” for family • Short term outcomes: • Better engagement in service delivery • Creative plans that fit the needs of youth/family • Improved service coordination • Follow-through on team decisions • Family regularly experiences success/support • Long term outcomes: • Stable, home-like placements • Improved mental health outcomes (youth and caregiver) • Improved functioning in school/ vocation and community • Improved resilience and quality of life • Model adherent wraparound • Youth/Family drives goal setting • Single, collaboratively designed service plan • Active integration of natural supports and peer support • Respect for family’s culture/expertise • Opportunities for choice • Active evaluation of strategies/outcomes • Celebration of success • Intermediate outcomes: • Achievement of team goals • Increased social support and community integration • Improved coping and problem solving • Enhanced empowerment • Enhanced optimism/self-esteem Phases and Activities of the Wraparound Process From Walker (2008)

4. In theory (i.e., when wraparound is implemented as intended), good things happen • High-quality teamwork and flexible funds leads to enhanced creativity, better plans, and better fit between family needs and services/supports • This in turn leads to greater relevance for families, less dropout • Strengths, needs, and culture discovery and planning process leads to more complete engagement of families • As family works with a team to solve its own problems, develops family members’ self-efficacy • Individualization and strengths focus enhances cultural competence, relevance, and acceptability • Focus on setting goals and measuring outcomes leads to more frequent problem-solving and more effective plans

5. In practice, however… W R A P A R O U N D ? ? ? • Until recently, there was no consensus on what wraparound consisted of • Growing evidence that much “wraparound” was not living up to its promise • This made it difficult to: • Coach, train, or supervise people with responsibility to carry out the wraparound process • Do quality assurance or develop evidence of wraparound’s effectiveness

6. A practice model:The Four Phases of Wraparound Phase1A Engagement and Support Phase1B Team Preparation Phase2 Initial Plan Development Phase3 Implementation Phase4 Transition Time

7. Phase 1 : Engagement and Team Preparation Care Coordinator & Family Support Partner meets with the family to discuss the wraparound process and listen to the family’s story. Discuss concerns, needs, hopes, dreams, and strengths. Listen to the family’s vision for the future. Assess for safety and make a provisional crisis plan if needed Identify people who care about the family as well as people the family have found helpful for each family member. Reach agreement about who will come to a meeting to develop a plan and where we should have that meeting. Phase 1 A and B

8. Phase 2: Initial Plan Development Conduct first Child & Family Team (CFT) meeting with people who are providing services to the family as well as people who are connected to the family in a supportive role. The team will: Review the family vision Develop a Mission Statement about what the team will be working on together Review the family’s needs Come up with several different ways to meet those needs that match up with the family’s strengths Different team members will take on different tasks that have been agreed to. Phase 2

9. Phase 3: Plan Implementation Based on the CFT meetings, the team has created a written plan of care. Action steps have been created, team members are committed to do the work, and our team comes together regularly. When the team meets, it: Reviews Accomplishments (what has been done and what’s been going well); Assesses whether the plan has been working to achieve the family’s goals; Adjusts things that aren’t working within the plan; Assigns new tasks to team members. Phase 3

10. Phase 4: Transition There is a point when the team will no longer need to meet regularly. Transition out of Wraparound may involve a final meeting of the whole team, a small celebration, or simply the family deciding they are ready to move on. The family we will get a record of what work was completed as well as list of what was accomplished. The team will also make a plan for the future, including who the family can call on if they need help or if they need to re-convene their team. Sometimes transition steps include the family and their supports practicing responses to crises or problems that may arise Phase 4

11. Positive Outcomes are Not Guaranteed! Wraparound teams often fail to: • Incorporate full complement of key individuals on the Wraparound team; • Engage youth in community activities, things they do well, or activities to help develop friendships; • Use family/community strengths to plan/implement services; • Engage natural supports, such as extended family members and community members; • Use flexible funds to help implement strategies • Consistently assess outcomes and satisfaction. • This is a problem, because better fidelity has been found to be associated with better outcomes

12. Implementation with High Fidelity Requires… County Context and Readiness State Support Staff Selection Organizational Supports Program Evaluation Training Supervision and Coaching Performance Management

13. Today’s training will focus on • An introduction to the TOM • History, purpose, psychometrics • User qualifications • Interviewer and observer training and supervision • Preparations to take before interviews and observations • Conducting TOM observations

14. Fidelity measurement • What is fidelity? • “The extent to which a treatment or intervention is delivered as intended” • In other words, “doing it right” • “Doing it right” in wraparound means: • Staying true to the 10 principles • Implementing the Phases and Activities

15. Ten Principles of the Wraparound Process • Family voice and choice. Family and youth/child perspectives are intentionally elicited and prioritized during all phases of the wraparound process. Planning is grounded in family members’ perspectives, and the team strives to provide options and choices such that the plan reflects family values and preferences. • Team based. The wraparound team consists of individuals agreed upon by the family and committed to them through informal, formal, and community support and service relationships.

16. Ten Principles of the Wraparound Process • Natural supports. The team actively seeks out and encourages the full participation of team members drawn from family members’ networks of interpersonal and community relationships. The wraparound plan reflects activities and interventions that draw on sources of natural support. • Collaboration. Team members work cooperatively and share responsibility for developing, implementing, monitoring, and evaluating a single wraparound plan. The plan reflects a blending of team members’ perspectives, mandates, and resources. The plan guides and coordinates each team member’s work towards meeting the team’s goals. • Community-based. The wraparound team implements service and support strategies that take place in the most inclusive, most responsive, most accessible, and least restrictive settings possible; and that safely promote child and family integration into home and community life

17. Ten Principles of the Wraparound Process • Culturally competent. The wraparound process demonstrates respect for and builds on the values, preferences, beliefs, culture, and identity of the child/youth and family, and their community. • Individualized. To achieve the goals laid out in the wraparound plan, the team develops and implements a customized set of strategies, supports, and services. • Strengths based. The wraparound process and the wraparound plan identify, build on, and enhance the capabilities, knowledge, skills, and assets of the child and family, their community, and other team members.

18. Ten Principles of the Wraparound Process • Persistence. Despite challenges, the team persists in working toward the goals included in the wraparound plan until the team reaches agreement that a formal wraparound process is no longer required. • Outcome based. The team ties the goals and strategies of the wraparound plan to observable or measurable indicators of success, monitors progress in terms of these indicators, and revises the plan accordingly.

19. TEAM OBSERVATION MEASURE • The FULL TOM user training presentation is divided into eight sections. • As mentioned earlier, these sections correspond to the eight chapters of the User’s Manual for the TOM • Each section of this training has a set of Learning Objectives that are presented at the beginning of the section • Learning objectives are also presented at the end of each section for review purposes • There are also several topics for group discussion related to local issues. These are indicated by Red text in the PowerPoint.

20. TOM Observer Training • Today we will cover the following sections: • An introduction to the TOM • History, purpose, psychometrics • Preparing to collect TOM data • Conducting TOM observations • Scoring rules for TOM indicators & items

21. The TOM • The TOM is designed to assess adherence to standards of high-quality wraparound during team meeting sessions . • It is organized according to the 10 Principles, with two items dedicated to each. • Each item has 3-5 indicators, which must be scored: • Yes (This was observed) • No (This was not observed) • N/A (This is not applicable)

22. Organization of the TOM • Cover Page • Observer records basic information about the meeting, and number and types of team members in attendance, and demographics. • The remaining pages present the 20 TOM items. • Indicators for each item are lettered from ‘a’ through ‘e’. • Total indicators = 71

23. Organization of the TOM – Response Scale • Response scale for indicators Each of the 71 TOM indicators must be scored as either ‘Yes,’ or ‘No.’ For some indicators, ‘N/A’ is an appropriate response. • Yes should be scored if, per the scoring rules and notes (provided in chapter 6), the described indicator was observed to have occurred during the meeting. • No should be scored if, per the scoring rules and notes, the described indicator was not observed to have occurred during the meeting. • N/A is an option for some items only, and is used if, for some reason, it is impossible to provide a score of Yes or No.

24. Organization of the TOM – Response Scale • Response scale for items: After scoring all the relevant indicators within an item, the observer must assign a score to the item as a whole. Each item includes a response scale from 0 – 4, whereby: • 0 = None of the indicators for this item were evident during the team meeting (i.e., none were scored ‘Yes’) • 1 = Some, but fewer than half of the indicators for this item were scored ‘Yes’ • 2 = About half of the indicators for this item were scored ‘Yes’ • 3 = More than half, but not all, of the indicators for this item were scored ‘Yes’ • 4 = All of the indicators for this item were evident during observation (i.e., all were scored ‘Yes’)

25. Organization of the TOM – Response Scale Note: WONDERS will calculate ITEM scores for you!

26. Role of the observers • The TOM observer is intended to be just that – an inconspicuous observer of the wraparound team process that occurs for a child and family along with her or his team members. • The observer should be well oriented to the TOM and the notes and scoring rules for each item and indicator that are presented in the chapter to follow. • The observer is expected to observe the entire team meeting, so that she or he can be certain whether the indicators did or did not occur during the meeting.

27. End of Chapter 1.Checking in / Discussion • Learning objectives: • Understand the organization of the TOM and its items • Understand the role of the observers in administering and scoring the TOM

28. Chapter 2.TOM User’s Qualifications • The Collaborator’s Manual specifies several necessary qualifications for the collaborating site and observers (we covered this in the WFI section) • Learning objectives: • Understand the expectations for a collaborating site • Understand the qualifications for observers

29. Observers training • Training and supervision should consist of: • An overview of the wraparound process, including its principles and four phases and activities; • An overview of the purpose & structure of the TOM; • A review of general TOM administration procedures; • A review of individual TOM items and scoring rules; • Group practice administrations of the TOM using a video taped meeting or approved live team meeting; • Practice administrations done in a pair with an experienced observer, evaluation leader, or supervisor, with comparison and de-briefing of scores assigned; and • Periodic group and/or individual supervision for observers.

30. Observers training • After initial group training and assessment of observer reliability and accuracy, it is recommended that one or more initial observation sessions with “live” teams be supervised by and evaluation team leader or supervisor. • Supervisor and observer should review scores assigned and discuss differences in scores and rationales for scores assigned, using the TOM manual. • Once the evaluation has begun, and observations are being conducted, team meetings or supervision sessions should also be held periodically so that members for the team can discuss administration issues they are encountering, scoring question, and other issues as a group.

31. End of Chapter 2.Checking in / Discussion • Learning objectives: • Understand the expectations for a collaborating site • Understand the qualifications for observers and observer training • DISCUSSION: • How will our evaluation team conduct practice observations be supervised? • Will we meet as a group over time?

32. Chapter 3.Preparing for TOM Observations • Preparation for conducting TOM observations requires preparation at several levels • Learning Objectives • To understand the requirements of any local IRB protocol • To be prepared to engage the different respondents in the TOM observation process • To select and prepare observers.

33. Before doing TOM Interviews • Project Approval • Are there any Institutional Review Board (IRB) or Human Subjects committee protocols for this evaluation? • Selecting & Preparing observers • It is important to use observers who are not directly involved with the services and supports that are being delivered to the families whose teams are being observed. • It is recommended that observers not personally know the participants. Anonymous observers who are not affiliated with members of families’ Wraparound Team are the best choice.

34. Before doing TOM observations • Approaching families and team members • Information about the TOM process must be provided to the family and wraparound facilitator or team leader. • Formal written or oral consent for their participation may also need to be obtained. Providing families with information about the evaluation process and TOM observations is crucial for ensuring they are fully willing and able to participate. • The evaluation should be presented as an opportunity for families to have their experience reviewed as a way to facilitate positive change in their community.

35. Before doing TOM observations • Engaging Facilitators and providers • As for caregivers and youths, facilitators (or care coordinators, or case managers, or team leaders) must be “on board” as stakeholders in the evaluation. • Facilitators and other team members need to be reminded that TOM data will be used to provide comprehensive (and confidential, in most TOM uses) feedback on how Wraparound practice is being implemented and that the data will be used to identify and support training needs.

36. End of Chapter 3.Checking in / Discussion • Learning Objectives • To understand the requirements of any local IRB protocol • To be prepared to collect complete data • To be prepared to engage the different respondents in the TOM observation process • DISCUSSION: What is the procedure in your program or site?

37. Chapter 4.Conducting TOM Observations • Before we get to administration and scoring rules for each TOM item, we must cover some basic instructions • Learning Objectives • To understand TOM ID numbers and how to track families • To understand rules for interview timing • To understand basic TOM issues: • Meeting types • Observation notes • Following up/debriefing • Scoring rules

38. Setting up for the meeting • Before you go to the team meeting, be sure you have all the materials you need. These materials may include: • Information on meeting location and time • TOM form and manual • An information sheet or evaluation project brochure to explain the TOM administration and evaluation to team members • Consent form(s), if required • Gift cards or other honoraria for participants, if being provided • Gift card receipts

39. Setting up for the meeting • Once you arrive, you should: • Introduce yourself and remind or explain to team members and other participants of the evaluation project’s purpose, if wraparound team facilitator/team leader has not done that • Have family sign Informed Consent Form, if necessary • Begin filling out cover sheet information

40. Meeting information • You will be asked to indicate which type of meeting you are observing. • This may be something that you know before you arrive at the meeting, or you may have to ask the wraparound facilitator or team leader which type of meeting is being conducted. • Please see the TOM manual for detailed descriptions of meeting types. DISCUSSION: Are there specific terms for meeting types in your program or site?

41. Completing the TOM • As a trained TOM observer, you should be prepared to be looking for information relevant to the 71 indicators on the TOM. • As the meeting progresses, take time to record your observations in the notes area to the right of each item. • You can also record your observations on a separate piece of paper or on the comments section on pages 7-8 of the TOM form • As things occur, you may also record your scores for relevant indicators by circling the appropriate response.

42. Observer notes and comments • Examples of why you scored “yes” or “no” for certain indicators • For example, if you score “yes” for indicator 8a (“Brainstorming of options and strategies include strategies to be implemented by natural and community supports”), in the “Notes” section, you might write: “8a Father mentioned that a neighbor had offered to teach the youth to drive. Team thought it was a good idea, and set this as one of the goals in the plan.”

43. Observer notes and comments • Non-verbal communication that clarifies scoring. • For example, if you score ‘No” for indicator 15b (“The team provides extra opportunity for the youth to speak and offer opinions, especially during decision making”) OR indicator 11a (“The team facilitator checks in with the team members about their comfort and satisfaction with the team process”), you might note in the “Comments” section: “During most of the meeting, youth sat at the table with arms folded and frowning. Appeared more and more upset as meeting progressed, but team did not check in with youth.” • OR this could be recorded in the “Notes” section for “Youth and family voice” (Item 15) or “Facilitation skills” (Item 11).

44. Types of comments and Notes NOT to include • DO NOT USE names. Use roles, job titles, or initials. • Do not give ONLY your opinions. Present specific evidence. For example, rather than writing, “The youth seemed angry,” say instead, “The youth sat the entire meeting he sat with a scowl on his face and his arms folded across his chest, and when he spoke his voice volume was loud and his voice tone was harsh.”

45. Scoring the TOM • After the meeting observation, plan on taking at least 30 to 60 minutes to sit down with your manual and TOM form to review your notes and complete your scoring while the meeting is still fresh in your mind. • For some of the TOM indicators, you may not have assigned a score; for others, you may feel the need to review your scores against the criteria in the manual. • Following up with the wraparound facilitator or team leader for certain indicators • As will be noted in the scoring rules for each indicator in the next chapter, it may be difficult to assign scores for some of the TOM indicators without additional information. • For these indicators, following up or debriefing with the team leader or facilitator may be necessary, either immediately after the team meeting, or on the phone at a later time, (if time does not permit an immediate de-brief).

46. Following up/Debriefs • TOM indicators for which a follow-up with the wraparound facilitator or team leader is permissible are marked with an asterisk on the TOM form. • For a complete summary of these items, see page 31 in the TOM Manual. DISCUSSION: What ways will your evaluation team ensure that facilitators/team leaders are receptive to follow-up questions from observers? When will this likely take place and how?

47. Scoring Rules • Different types of meetings consist of different types of content. • E.g., Follow-up meetings that are taking place many months after the plan of care was developed may present less information about the TOM indicators than a planning meeting. • Nonetheless, remember that objective information must be the basis for all scores assigned. This primarily will consist of behaviors observed in front of the observer in a meeting. • Though you may follow up with the wraparound facilitator or team leader or review the plan of care to score a few specific items, as an observer, you should rely primarily on what you see in the meeting.

48. End of Chapter 4.Checking in / Discussion • Learning Objectives • To learn how to set up for a meeting • To understand basic TOM issues: • Meeting types • Observer notes • Following up/debriefing • Scoring rules

49. Wraparound Projects (N=6) with coaching to certification: Mean WFI scores

50. Sites with both WFI-4 andTeam Observation Measure (TOM) Scores Site 1 WFI n=19 / TOM n=4 Site 6 WFI n = 22 / TOM n=13 Site 3 WFI n=19 / TOM n=14 Site 7 WFI n = 3 / TOM n=3 Site 5 WFI n=17 / TOM n=10 Site 8 WFI n = 50 / TOM n=24 Site 9 WFI n=110 / TOM n=39 Site 10 WFI n = 207 / TOM n=16