Adjusting our FOCUS! Measuring Meaningful Clinical Outcomes

1. F Focus on the O Outcomes of C Communication U Under S Six Adjusting our FOCUS!Measuring Meaningful Clinical Outcomes Paediatric Symposium '11 Building a Brighter Future

2. Team FOCUS • Prof. Nancy Thomas-Stonell, PI • Dr. Bruce Oddson, Co-PI • Dr. Peter Rosenbaum, Co-PI • Dr. Karla Washington, Post-Doctoral Fellow • Ms. Bernadette Robertson, Research Coordinator • Ms. Joan Walker, Research Assistant Funding SickKids Foundation Canadian Institutes of Health Research (CIHR) Bloorview Childrens Hospital Foundation Paediatric Symposium '11 Building a Brighter Future

3. Eleven Research Partners in Five Provinces Across Canada • Eastern Healthcare, St. Johns, NL • Nova Scotia Hearing and Speech Centres, NS • Beyond Words, Preschool Speech and Language Program, York Region, ON • Waterloo Preschool Speech and Language Program, ON • Holland Bloorview Kids Rehabilitation Hospital, ON • Hamilton Preschool Speech and Language Service, ON • Technology Access Clinic, ON • Wellington-Dufferin Guelph Region Preschool Sp & Lang Service, ON • ErinoakKids Centre for Treatment and Development & Halton-Peel Preschool Speech and Language Program, ON • Calgary Health Region, AB • BC Centre for Ability, BC Paediatric Symposium '11 Building a Brighter Future

4. A Typical Day in the Life of a SLP! Example prepared with thanks to Laurie Graham Paediatric Symposium '11 Building a Brighter Future

5. Preschool Speech-Language Pathologist 4 year old boy, A.B., presents for service Formal assessment results (currently available standardized tests) indicate a moderate speech and language disorder Paediatric Symposium '11 Building a Brighter Future

6. Assessment reveals… • Has a lot of trouble with sounds • i.e. ‘ish’ instead of ‘fish’; ‘ • og’ instead of ‘frog’ • Has trouble with pronouns • i.e. often says ‘he’ instead of ‘she’ Paediatric Symposium '11 Building a Brighter Future

7. Parents reports the boy is: extremely frustrated, has tantrums teased at school kids and teacher have trouble understanding him shy, evidence of low self-esteem Paediatric Symposium '11 Building a Brighter Future

8. Clinical Goals for 8 week treatment block A.B. will produce /f/ in word initial position in structured settings 80% of the time. A.B. will produce ‘she’ appropriately in phrases in structured settings 80% of the time. Speech-Language Pathologist documents parent comments in client file. Paediatric Symposium '11 Building a Brighter Future

9. Eight weeks later… During the last session, the Speech-Language Pathologist decides her goals have not been met as the child is only performing tasks at a 50% success rate. The parents state that tantrums have decreased, A.B. is no longer being teased at school, and seems more confident in communicating. Paediatric Symposium '11 Building a Brighter Future

10. The importance of parent comments The Speech-Language Pathologist has chosen to include many parent comments in her client notes because they give her an indication of the child’s interactions with others. Paediatric Symposium '11 Building a Brighter Future

11. The importance of parent comments • Interaction is fundamental to the development of communication - the more you interact, the more you practice communication skills. • The ability to communicate with peers and teachers is fundamental to academic and social success (i.e. group work). Paediatric Symposium '11 Building a Brighter Future

12. Parent comments continued.. Including parent comments in client notes, although recommended by regulatory bodies of the profession, is not required. There are no valid and reliable measures (i.e. tests/questionnaires) that capture the real-world changes observed by parents and clinicians! Paediatric Symposium '11 Building a Brighter Future

13. Outcome measurement for A.B. The Speech-Language Pathologist wishes she could more thoroughly document the behavioural, interactive and social changes seen by A.B.’s parent. She suspects that A.B.’s gains are associated with therapy (not just normal development) but there is no way to prove her hypothesis. Paediatric Symposium '11 Building a Brighter Future

14. Decision time: There are other children on the waitlist. Given the limited funding available, A.B. is not offered a second block of therapy. Paediatric Symposium '11 Building a Brighter Future

15. Increasing concern Clinicians, researchers and disability advocates are concerned that changes which may be important results of therapy are overlooked. Paediatric Symposium '11 Building a Brighter Future

16. What are clinically meaningful outcomes? Paediatric Symposium '11 Building a Brighter Future

17. World Health Organization (WHO) Health Frameworks • In 1980 WHO (1980) came out with their first health framework the International Classification of Impairments, Disabilities and Handicaps (ICIDH) • Impairment (what’s broken) • Disability (what can’t you do) • Handicap (limitations in the real world) Impairment Disability Handicap Paediatric Symposium '11 Building a Brighter Future

18. International Classification of Functioning, Disability and Health (ICF-2003)Children & Youth Version (ICF-CY - 2007) Health Condition Body Functions & Structures Activities Participation Environmental Factors Personal Factors Paediatric Symposium '11 Building a Brighter Future

19. ICF & ICF-CY Domains • Body Functions: Physiological • (e.g., voice, oral motor, speech production) • Body Structures: anatomical • (e.g., hearing loss; CL/P) Paediatric Symposium '11 Building a Brighter Future

20. ICF & ICF-CY Domains • Activities: • Tasks and actions by an individual. • ‘Capacity’ - performance of a task in a standard environment. • Participation: • Involvement in a life situation. • ‘Performance’ performance of tasks in a in the current environment. Paediatric Symposium '11 Building a Brighter Future

21. Contextual Factors • Environmental Factors: • external influences on functioning and disability related to physical, social and attitudinal world. (stairs, culture, support system) • Personal Factors: • internal influences on functioning and disability (personality influences on coping style) Paediatric Symposium '11 Building a Brighter Future

22. ICF Health Framework • Has positive and negative components. • Uses a bi-directional model. • Doesn’t take developmental stages into account. • e.g. temper tantrums/frustrations for 2 year olds Paediatric Symposium '11 Building a Brighter Future

23. ICF-CY Health Framework • New codes to capture the functional characteristics of a developing child. Expanded codes include: • Learning new skills • Play • Adaptability • Persistence • Exploration Paediatric Symposium '11 Building a Brighter Future

24. Why use ICF-CY model? • Outcomes need to be evaluated across ICF-CY domains. • Several studies (Sarno, 1969, Aten, 1986) have noted poor correlation between body structure/function outcomes and activity and participation outcomes • depends not only on skill levels, but also personality, coping skills, social support systems... Paediatric Symposium '11 Building a Brighter Future

25. How do we measure these outcomes? Paediatric Symposium '11 Building a Brighter Future

26. We need treatment outcome measures! We need outcomes measures to evaluate the impact of treatment on children’s lives. Paediatric Symposium '11 Building a Brighter Future

27. Outcomes vs. Outcome Measures • Any consequence of healthcare is an outcome! Outcome = environment + treatment + client + severity Paediatric Symposium '11 Building a Brighter Future

28. There are many types of outcomes. • Avoiding adverse affects of care (nobody dies) • Improved physiologic status ( phonation time) • Reduction in symptoms ( dysfluencies) • Improved functional status (use telephone) • Minimizing costs • Minimizing length of care Paediatric Symposium '11 Building a Brighter Future

29. Outcome Measure • A treatment outcome measure is a validated test designed to measure change in function. • It measures, in quantitative terms, the impact of routinely delivered care on client’s lives. Paediatric Symposium '11 Building a Brighter Future

30. Treatment Outcome Measures need to be proven to work! • Garbage in – garbage out! • Information generated by outcome studies is only useful if the measure is clinically useful and scientifically sound (van der Putten et al., 1999). Paediatric Symposium '11 Building a Brighter Future

31. Why can’t we use our standardized tests? Paediatric Symposium '11 Building a Brighter Future

32. Standardized tests... • Determine the presence or absence of a communication disorder. They do not change. • They provide too little information (insufficient number and variety of items) to monitor progress. Huang, Hopkins & Nippold (1997). Satisfaction with Standardized Language Testing: A survey of Speech-Language Pathologists. Language Speech & Hearing Services in Schools 28, 12-29. Paediatric Symposium '11 Building a Brighter Future

33. Treatment Outcome measures • Outcomes measures at a minimum need to be proven to reliably distinguish between children who improve from therapy and those who do not improve. Paediatric Symposium '11 Building a Brighter Future

34. Why use treatment outcome measures? • To improve treatment services in an evidence-based manner. • To measure clinically important change. • To determine optimal length for treatment. • To select the best treatment approach for each child. Paediatric Symposium '11 Building a Brighter Future

35. CASLPA Position Statement on Outcome Measures CASLPA encourages and supports the use and development of outcome measures by speech-language pathologists and audiologists Outcome measures should be used to improve practice in an evidence-based manner in the best interests of clients. CASLPA Position Statement on Outcome Measures – May, 2010 (www.caslpa.ca) Paediatric Symposium '11 Building a Brighter Future ASHA Convention November 18-20 2010 35

36. The FOCUS journey began in 1998 • Holland Bloorview Kids Rehabilitation Hospital wanted an outcome measure for speech-language therapy that could be used across programs. • Diverse population • CP/CLP/ABI Paediatric Symposium '11 Building a Brighter Future

37. Began our search for a treatment outcome measure. • TOMS and AusTOMS are very broad measures of change. Scale has many descriptors. Hard to know what changed. • GAS (individualized and time consuming). • ASHA NOMS had no proven reliability or validity. We completed a two-year study to evaluate the NOMS. Results indicated poor sensitivity to change. Paediatric Symposium '11 Building a Brighter Future

38. What do we do now? • …with no existing valid, reliable and responsive communication outcome measure for preschool children available? Paediatric Symposium '11 Building a Brighter Future

39. Development of the ‘FOCUS’ • F ocus on the • O utcomes of • C ommunication • U nder • S ix • Thomas-Stonell, N., Oddson, B., Robertson, B. & Rosenbaum, P. Development of the FOCUS (Focus on the Outcomes of Communication Under Six), a communication outcome measure for preschool children. Developmental Medicine and Child Neurology: 2010, 52:47-53. ] Paediatric Symposium '11 Building a Brighter Future

40. Our Goal To develop a valid,reliable,responsive treatment outcome measure that captures ‘real world’ changes following speech and language treatment. Paediatric Symposium '11 Building a Brighter Future

41. Developing the FOCUS • In the previous outcome study, we collected data from parents of 210 preschool children receiving speech-language treatment and their clinicians. (Thomas-Stonell, Oddson, Robertson & Rosenbaum, Predicted and Observed Outcomes in Preschool Children Following Speech and Language Treatment: Parent and Clinician Perspectives. JCD 42 (2009) 29-42.) Paediatric Symposium '11 Building a Brighter Future

42. Developing the FOCUS • They were asked to describe the changes they observed in their child during/following therapy. • My child is now able to… • What other changes did you see? • Why is that important? Paediatric Symposium '11 Building a Brighter Future

43. Method: 6 Linked-Steps • Content analysis of our descriptive data to create FOCUS items. 2. Test the measure with clinicians and families. 3. Revise the measure using the parent and clinician feedback. 4. Test the revised measure again. 5. Revise measure a second time. 6. Test measure a third time to obtain preliminary reliability and validity data. Paediatric Symposium '11 Building a Brighter Future

44. Content Analysis • Content analysis is the “systematic, objective analysis of message characteristics” to make valid inferences from text. (Neuendorf, 2002) Paediatric Symposium '11 Building a Brighter Future

45. Content Analysis • Identify recurring categories of change and calculate percentages of occurrence for each category. • The recurring categories reflected the ICF-CY framework. Paediatric Symposium '11 Building a Brighter Future

46. Coding Comments Working Slides Paediatric Symposium '11 Building a Brighter Future

47. Developing the FOCUS • FOCUS is driven by DATA, • no preconceived ideas • FOCUS items were developed from categories cited by >10% of parents & clinicians. • Resulted in 200 items, reduced to 103. • Items used parents’ own wording. Paediatric Symposium '11 Building a Brighter Future

48. Sample Body Functions Item Development Paediatric Symposium '11 Building a Brighter Future

49. Item Development: Body Functions Parent Comment “Pronounces words much more clearly (specifically F sounds,L sounds when prompted - he still has work to do with L's)” Category/ICF-CY coding: Body Functions • Articulation Functions; b320 FOCUS Item “My child’s speech is clear.” Paediatric Symposium '11 Building a Brighter Future

50. Sample Activities/CapacityItem Development Paediatric Symposium '11 Building a Brighter Future