Child and Family Outcomes: Data Workshop

1. Child and Family Outcomes:Data Workshop Siobhan Colgan, ECO at FPG Melissa Raspa, ECO at RTI Cornelia Taylor, ECO at SRI Improving Data, Improving Outcomes Conference Washington, DC September 16, 2013

2. Session Overview • Family outcomes data drill-down • Data quality: Response rates, representativeness • Pattern checking: Outcomes by program, race/ethnicity, and disability type • Child outcomes data drill down update this? • Data quality: Missing data, categories a and e • Pattern checking: Outcomes by program, race/ethnicity, and disability type • Linking child and family outcomes Early Childhood Outcomes Center

3. But First… • Who are you • Part C or Part B 619 • Coordinator, data manger, other • More familiar with • Family or child outcomes • Comfort-level with data and analyses • I’m raring to go! • I need a warm-up • Help! Early Childhood Outcomes Center

4. The Big Picture of Program Improvement Early Childhood Outcomes Center

5. The Big Picture of Program Improvement • Are we where we want to be? • What issues do we want to address? • Did we accomplish what we intended? • What do we need to do to get there? • Make planned changes Early Childhood Outcomes Center

6. Guiding Program Improvementwith Critical Questions • Asking critical questions can help you • Reflect on current status • Plan for possible changes • Check to ensure implementation is working • Critical questions should drive your analyses • Questions can come from a variety of sources or stakeholders Early Childhood Outcomes Center

7. From Critical Questions… to Root Causes What are the numbers? How do you interpret the numbers? What should you do? Early Childhood Outcomes Center

8. Welcome to Euphoria! • Part C program serves approximately 8,000 children a year • 10 regional programs of varying sizes • Family outcomes survey is sent to half the families each year Early Childhood Outcomes Center

9. Steps to Data Analysis • Step 1: Data Quality • Examine data quality to have confidence in your analysis results • Gives you confidence that data are “real” • Step 2: Pattern Checking and Drill Down • Examine specific questions related to your data • Results can be used to make decisions about program improvement Early Childhood Outcomes Center

10. Family OutcomesStep 1: Data Quality Check • Response rates • Did everyone who was supposed to respond to the survey actually respond? • Proportional representativeness • How close do the response rates match the comparison data?

11. Response Rates • Possible categories to use for comparison • Race/ethnicity • Region or program • Age • Disability type • Length of time receiving services • Suggestions for others? • Income • Medicaid status

12. Example 1 Response Rates • CRITICAL QUESTION • How do our response rates vary by race/ethnicity? • Are certain types of families more likely to respond to the survey? • ANALYSIS PLAN • Compare families who received the survey to those who responded to the survey using race/ethnicity

13. What is the overall response rate? How do the response rates compare across groups? Are any groups over- or under-represented in the data?

14. Proportional Representativeness • Possible data used for comparison • Part C population (i.e., child count) • 618 data • Target population (e.g., all those exiting program) • State data • Suggestions for others? • Sampling plan

15. Example 2Proportional Representativeness • CRITICAL QUESTION • Are the families who responded comparable to those in early intervention state-wide? • Are all types of families represented in the data? • ANALYSIS PLAN • Compare families who responded to the survey to child count data

16. What data are being used for comparison? How do the percentages compare across groups? Where any groups over- or under-sampled? Are the data representative across groups?

17. Family Outcomes Step 2: Pattern Checking • Answer specific questions in order to make decisions about program improvement • Family outcomes by • Program • Race/ethnicity • Severity level • Others? • Home language • Family SES Early Childhood Outcomes Center

18. Example 3 Family Outcomes by Program • CRITICAL QUESTION • Do some of our programs serve families more effectively than others? • ANALYSIS PLAN • Compare percentages of families who met the indicators across programs Early Childhood Outcomes Center

19. Outcomes by Program Early Childhood Outcomes Center

20. Discussion Questions • Looking at the 3 family outcomes by program • What is the overall pattern across the outcomes? • Does this match the state as a whole? • Which programs are struggling on each of the outcomes? • Which programs are performing better? Early Childhood Outcomes Center

21. Example 4Family Outcomes by Race/ethnicity • CRITICAL QUESTION • Do some of our programs serve certain types of families more effectively than others? • ANALYSIS PLAN • Compare percentages of families who met the indicators across race/ethnicity categories Early Childhood Outcomes Center

22. Outcomes by Race/Ethnicity Early Childhood Outcomes Center

23. Discussion Questions • Looking at the 3 family outcomes by race/ethnicity • Which types of families are struggling on each of the outcomes? • Which types of families are faring better? Early Childhood Outcomes Center

24. Example 5Family Outcomes by Severity Level • CRITICAL QUESTION • What critical question is this chart answering? • Do some of our programs serve families with certain types of children more effectively than others? • ANALYSISPLAN • What is the analysis plan for this question? • Compare percentages of families who met the indicators across race/ethnicity categories Early Childhood Outcomes Center

25. Example 5Family Outcomes by Severity Level Early Childhood Outcomes Center

26. Discussion Questions • Looking at the 3 family outcomes by severity level • How are families with mildly affected children doing compared to families with more severely affected children? Early Childhood Outcomes Center

27. Example 6Knowing Rights Drill Down • CRITICAL QUESTION • Based on data in example 3, we know some programs are still struggling with Knowing Rights. Which groups of families within these programs have lower outcomes? • ANALYSIS PLAN • Within programs, examine race/ethnicity by Knowing Rights Early Childhood Outcomes Center

28. Knowing Rights by Program by Race/ethnicity Early Childhood Outcomes Center

29. Discussion Questions • Within each program • How are different types of families doing? • Are the same types of families struggling in both these programs? • Is the pattern similar to the state-level data? Early Childhood Outcomes Center

30. Break 15 minutes! Early Childhood Outcomes Center

31. Linking Child and Family Outcomes Data • Why link data? • In Part C, working with children and families • Integrate outcomes into IFSP/IEP process • Answer critical questions that put child in context of family Early Childhood Outcomes Center

32. Helping Families, Helping Children Early Childhood Outcomes Center

33. Pros and Cons of Linking Outcomes Data • Pros • Big picture of how state and programs are serving children and families • Answer critical questions; able to drill down to see patterns in the data • Plan for purposeful program improvement • Cons • It’s hard! • Different systems need to talk to each other (i.e., need a common identifier) • Difficult to think through how family and child outcomes should be related Early Childhood Outcomes Center

34. Welcome to Euphoria! • Euphoria is able to link child and family outcomes using a common identifier • Want to look at relationship between how well families are achieving outcomes and child progress Early Childhood Outcomes Center

35. Example 7Linking FO3 and CO2 • CRITICAL QUESTION • Do families who know how to help their child develop and learn have children who make more progress on acquiring language and skills? • ANALYSIS PLAN • Of families who meet FO3, compare differences between categories b and c/d on CO 2 Early Childhood Outcomes Center

36. How do families compare on helping their child develop and learn across progress categories on CO2? • Is this relationship expected? Early Childhood Outcomes Center

37. Example 8Linking FO2 and CO3 • CRITICAL QUESTION • Do families whose children make little or no progress know how to communicate their children’s needs better than families whose children come in at age expectations when it comes to using behavior to meet their needs • ANALYSIS PLAN • Of families who meet FO2, compare differences between categories a and e on CO3 Early Childhood Outcomes Center

38. How do families compare on communicating their child’s needs across progress categories on CO3? • Is this relationship expected? Early Childhood Outcomes Center

39. Early Childhood Outcomes Center

40. Example X: Family Outcome Trends • CRITICAL QUESTION • How do our family outcomes data look over time? • ANALYSIS PLAN • Compare percentages of families who met each indicator over the past 3 years Early Childhood Outcomes Center

41. Trends over Time • For FFY11, how is this state doing on each of the 3 indicators? • Looking at the last 3 years, what can you say about this state’s C4 data? Early Childhood Outcomes Center

42. Comparison to National Data • How does this state compare to the national averages? Early Childhood Outcomes Center