The Child and Family Services Review

1. The Child and Family Services Review May 1, 2008

2. The Child and Family Services ReviewAugust 2007

3. The Child and Family Services Review • What the Child and Family Services Review (CFSR) is NOT: • An audit of the child welfare system • A statistically valid assessment of child welfare outcomes • A complete and comprehensive assessment: • No review of 74 counties • No comparative data to demographically comparable states • The assessment of the judicial system is limited to 3 counties • The assessment of the service delivery systems for health, education and mental health services needed to support children and families is also geographically limited.

4. The Child and Family Services Review What the Child and Family Services Review (CFSR) IS: • A PROCESS that assesses improvement • A Continuous Quality Improvement (CQI) Process • Inclusive of the Courts, Attorneys, Education, Medical providers – not limited to the OKDHS • Measures trends of practice and improvements: Systemic Measures; Outcome Measures. • A detailed review of 65 cases from 3 counties • Administered by the U.S. Department of Health and Human Services, Administration on Children, Youth and Families, the Children’s Bureau

5. The Child and Family Services Review • Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Strengths of the CFSR Process: • Even though Outcome Measures from cases involve such small numbers that they are not statistically supportable, lessons can be learned that indicate improvements are worthy of serious evaluation, even though caution must be exercised in determining what improvements are appropriate

6. The Child and Family Services Review • Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Strengths: • Some assessments (systemic ones) have more credibility after interviewing dozens of stakeholders • Other assessments (outcome measures ) have less credibility because of the absence of statistical validity • Systemic assessments measure processes, i.e. the presence of quality data systems; a continuous quality improvement process, etc. • Outcome measures measure individual case results • It is a detailed review, using only 65 cases: • 31 Oklahoma County; 17 Muskogee County; 17 Comanche

7. The Child and Family Services Review • Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Weaknesses: • The number (the “n”) of cases reviewed is so small many of the conclusions are not statistically supportable • Even with the limited number of cases, several cases are excluded for some of the outcome measures because the outcome is not applicable to the selected case, (e.g. the case may measure the timeliness of an investigation, but if the child was not removed, the case would not be counted for any foster care outcomes because the child never entered foster care). This exclusion of cases for some measures makes the “n” even smaller and the statistical validity even more remote • Sixty-five cases assessed; but for some standards only 40 cases are used or other standards only 25 or even fewer cases are used • Therefore failing to meet the standard on 3 or 4 cases of 15 or 40 may mean an outcome is not met for the entire state.

8. The Child and Family Services Review • Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Weaknesses: • The standard is also unrealistically high – 95 percent of the reviewed cases must be rated as a strength for a finding of “substantially achieved” on the outcome. • If all 65 cases are used on an outcome, only 3 cases can be rated lower than a strength • If only 20 cases are used, all cases must be rated as a strength • Nevertheless, the process of improvement is the goal, not the score

9. The Child and Family Services Review Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Weaknesses: • A poor assessment on 3 or 4 cases in the outcomes section gives the appearance that the entire state needs improvement • A strong assessment on 10 to 15 cases might mask a need for improvement • Because the “n” is so small it is likely that regional variations will be found and may be wide, but not necessarily statistically supported

10. The Child and Family Services Review • Strengths and Weaknesses of the Child and Family Services Review (CFSR) • Weaknesses: • Very misunderstood tool for improvement • No state has ever “passed” a CFSR • It is unlikely that any state will ever “pass” a CFSR since the “National Standard” uses data that has serious comparability flaws and the “n” used to assess a state’s compliance is so small, the statistical probability of “passing” is very small. • NEVERTHELESS, the process for improvement is worthy of the effort

11. The Child and Family Services Review Status of the Child and Family Services Review (CFSR) • State visit was in August 2007 • Report should be finalized in March 2008 and will be on the Children and Family Services Division Web site • The state will have 90 days to prepare a Program Improvement Plan (PIP) which must be approved by representatives of the Federal government • Compliance with the PIP will then be monitored by representatives of the Federal government. • The process is a Continuous Quality Improvement Process

12. The Child and Family Services Review Summary self-assessment of our Child Welfare system: • System Strengths: • Data systems: • First federally approved Statewide Child Welfare Information System in the nation • One of only 6 in the nation • The electronic file is usually the working file • Monitoring Child Safety: • According to an HHS Office of Inspector General study in December 2005, Oklahoma was 1 of only 5 states (CO, UT, SC and FL) who could document that more than 90 percent of our foster children received a monthly visit from a Child Welfare staff person

13. The Child and Family Services Review Summary self-assessment of our Child Welfare system: • System Strengths: • Committed to improvement; CQI processes • Many recent innovations: • Fostering Hope Clinic was started in OKC about two years and recently added a presence in Tulsa • Designation of Dr. Deb Shropshire as CW Medical Director • Only known co-located Child Support Enforcement office within a Child Welfare office in the nation; Helps identify non-custodial parents; Helps establish child support while in care • OMI federal grants for foster parents and adoptive parent couples retreats

14. The Child and Family Services Review Summary assessment of Oklahoma’s Child Welfare system: • Outcome Strengths: • Oklahoma average length of stay in foster care: 21.2 months; compared to a national average of: 28.8 months • Record numbers of children being adopted out of foster care; one of the highest per capita rates in the nation. • Low rates of children re-entering foster care • High rates of children being placed in close proximity to their parents • High success of meeting children’s physical health needs

15. The Child and Family Services Review Summary self-assessment of Oklahoma’s Child Welfare system: • Where system improvement is needed • Staffing: • Our intake and investigation are generally adequately staffed; but permanency planning staff need additional workers to make the caseloads manageable • However, staffing does not always translate into shorter Average Length of Stay in care; but probably will reduce turnover and make the job less “reactionary.” • Case review process • Uneven practice across the state • Different courts, prosecutors and social workers have mixed experience reviewing cases • Quality of assessments in cases can be improved

16. The Child and Family Services Review Summary self-assessment of our Child Welfare system: • Where system improvement is needed • More Foster Care Homes • One strength is that a significant number of our children are placed with relatives or kin • Keeps children connected to their family and friends • Kin usually are more committed and have fewer disruptions • Many foster families adopt their foster children, therefore, replacement of the foster homes must be continuous • Replacing non-kinship foster homes is about flat while the number of children coming into care has gone up over the last few years leaving a shortage of traditional foster care homes. • Contracts with Emergency Foster Care providers have had only marginal success, even though rates are higher

17. The Child and Family Services Review Summary self-assessment of our Child Welfare system: • Where Outcome Improvement is needed • Placement Stability: • Average child in Oklahoma experiences 2.9 placements while in care; about the regional average • More than Colorado (2.1) and New Mexico (2.3) • Less than Texas (3.5) and Kansas (4.1)

18. The Child and Family Services Review CAUTION about using data in Child Welfare systems: • Using the main data elements are fine • For Example: • Average length of stay of children in care or leaving care; Placement stability; Length of time to permanency • Other data elements should be treated cautiously: • Child Welfare League of America has detailed more than 100 pages of definitions to promote consistency between states on data, but cautions that comparability is difficult for many reasons: • Different states have different definitions • Different states have different monitoring systems • Different states have varying degrees of mature data systems and a few states have very modest systems

19. The Child and Family Services Review How does Oklahoma’s CFSR Scores Compare to other states: • There are 20 scores • Six pre-site visit scores • Seven site visit “outcome scores” • Seven site visit “process scores • Only 7 states have received scores on the second round of the Review • Of the 7 states receiving scores to date, Oklahoma’s scores are neither the best nor the worst in any category • Oklahoma passed 5 of 7 site visit “process scores” which tied Oklahoma for second among the other 6 states receiving scores to date.

20. Ways to Assess and Improve the Child Welfare System

21. Ways to Assess and Improve the Child Welfare System • Assess Performance where statistically valid: • Measured against prior period performances • Measured against comparable data in other states • Assess Processes: • Frequency of documenting certain activities • Thoroughness of data collection and retrieval • Timeliness of completing core tasks • Participating in a Continuous Quality Improvement Process • Timeliness and adequacy of Case Review process

22. Ways to Assess and Improve the Child Welfare System • Assess Outcomes • Average length of stay in out of home care • Average number of placements in foster care • Prevalence of repeat maltreatment

23. Ways to Assess and Improve the Child Welfare System • How do we assess and improve the performance of the Child Welfare system? • OKDHS maintains a Quality Improvement unit in the Children and Family Services Division: • Work with each county on system improvement plans • Develop our state’s Program Improvement Plan (PIP) to report to the Federal Government • The Court Administrator’s Office of the Oklahoma Supreme Court maintains staff who work on the Court Improvement Program.

24. Ways to Assess and Improve the Child Welfare System • How do we assess and improve the performance of the Child Welfare system? • OKDHS works with researchers from the OUHSC and the University of Kansas using OK data to help develop improved practices. • OKDHS tracks key indicators by office: • Average length of stay in foster care • Average length of stay of children exiting foster care • Authorized and finalized adoptions • Timeliness of investigations of allegations • Documented frequency of visiting foster children

25. Ways to Assess and Improve the Child Welfare System • How do we assess and improve the performance of the Child Welfare system? • OKDHS invests in training staff and participating in national organizations: • Child Welfare League of America (CWLA) • National Children’s Alliance (NCA) • The American Public Human Services Association’s (APHSA) National Association of Public Child Welfare Administrators (NAPCWA)

26. Ways to Assess and Improve the Child Welfare System How do we assess and improve the performance of the Child Welfare system? • OKDHS participates in the federal assessment called the Child and Family Services Review (CFSR)

27. Next Steps

28. The Child and Family Services Review Next Steps • The next step in the Continuous Quality Improvement Process is the development of a Program Improvement Plan • Oklahoma will have 90 days to prepare a Program Improvement Plan (PIP) which must be approved by representatives of the Federal government • Compliance with the PIP will then be monitored by representatives of the Federal government.