Challenges of Evidence-Informed Out-of-Home Care Policy: The Role of Systematic Mapping

1. The challenges of being evidence-informed in out-of-home care policy and practice: exploring the potential of systematic maps Belinda Mayfield, BSW, PhD

2. Background/Contextual factors • AASW code of ethics – to maintain and expand levels of current knowledge, theory and skill • Barriers to accessibility of knowledge sources from the professional literature; time, oral culture, short timeframes for decisions, skill in searching, cost, gaps in research, debates about ‘evidence’ • Audit of Australian out of home care research (Cashmore & Ainsworth, 2004) 1995-2004 • Crime Misconduct Commission Inquiry (2005) • Out-of-home care: Messages from research(Bromfield,Higgins,Osborn,Panozzo & Richardson, 2005) • Emergence of systematic mapping to systematically and transparently describe the extent of research in a field, to identify gaps and to provide direct links to the evidence (Clapton et al., 2009)

3. Research aims Overall To investigate the issues of the accessibility of the out-of-home care literature as a source of knowledge to professionals in roles which influence policy and practice. Specifically, the aims were to: • Develop a systematic map of the out-of-home care literature • Provide a descriptive analysis of the composition of the available literature to identify gaps and areas of substantial knowledge • Gain insight about how different user roles view their knowledge needs and the barriers to accessibility • Gather feedback about the utility and limitations of the map

4. Why a map and not a systematic review • In social sciences interest in not only what works but for whom, in what circumstances, why and for what outcomes • Practitioners perspective – case management includes key decisions about removal, assessment of protective and care needs, placement matching, interventions etc. – broader scope than a review • Policy makers – often short timeframes eg post inquiry 12 weeks to prepare the Blueprint for reform; systematic reviews narrowly focused and 6 months + to undertake • Program development and management require different types of information, re cost effectiveness, feasibility, service and system level outcomes; new initiatives not yet evaluated • How theory, policy and practice literature may also inform….rather than directing what we do - to query/challenge current jurisdiction specific approaches; have we got it right or are there other ways eg permanency

5. Overview of study with 2 key phases: • A systematic map of out-of-home care literature 1995-2008 • Interviews with “users” to examine the utility of the map. User perspectives include: • Researchers & knowledge ‘translators’ • Policy makers, program managers & practitioners

6. Conceptual framework • Ecosystemic view of out-of-home care – built on Australian audit • Case management framework • Different types of knowledge, their interrelationships ( complementarities, inconsistencies and gaps), multiple uses • Models of knowledge utilisation

7. Model of Professional Knowledge (Drury-Hudson, 1997) Theoretical Knowledge Empirical Knowledge Procedural Knowledge Personal Knowledge Professional Knowledge Practice Knowledge

8. Accessibility - Exposure phase of Knowledge utilisation

9. Initial planning, topic setting and preliminary work Establish inclusion/exclusion criteria Searching Screening and coding Analysis and development of map report Finalisation of the systematic map Dissemination Methodology of systematic mapping

10. Methodology • Preliminary handsearch of a key UK journal and USA journal • Electronic searches to test individual search terms, prior to a ‘rolled up’ search • 1995-2008 ( built on timeframes for Australian audit) • Searches in PsycInfo, ERIC, Social Services Abstracts,Medline,Family and Society Plus.

11. Search strategy

12. Findings • Content findings • Methodological findings • Knowledge needs of different users • Barriers and enablers to accessibility of the out-of-home care literature

13. Total reports identified n=12,939 (3,229 duplicates) Research studies n=2,616 Conceptual or theoretical n=981 Total abstracts screened n=9,710 Literature reviews n=207 Program description n=387 Screened In N=4,425 Screened out n=5,285 Book reviews n=234 Insufficient data to screen n = 945 Not child protection population n=694 Not specific to out-of-home care n=2,986 Not on topic n=660 Risk assessment n=146 Family preservation n=110 Prevention and early intervention n=137 Overall screening results

14. Overall composition of the literature

16. Trends in publications reporting research

17. Number of publications by jurisdiction

18. Trend in publications reporting literature reviews

19. Analysis of composition of content areas • Examined 6 overarching content areas: • Families of children in care • Children and young people in care • Foster families • Kinship care • Residential care • Quality of care

20. Analysis • Composition of the literature • Gaps and areas for further investigation: • Research gaps • Literature reviews

21. Example

22. Methological findings • Volume and complexity in locating relevant references (over 9,500 references= over 4,000 relevant publications and 2,615 research literature • Time consuming task – approx 50 hours to search; screening and coding (approx. 1 hour to screen 20 abstracts) Total approx. 500 hrs • 62% of research found in PsychInfo • 38% of research across other 4; 5% unique to Medline • Approx double number of Aust. studies identified by audit • Analysis phase 8-12 weeks

23. Informants views • Knowledge needs: • Research • Theory • Practice • Strategies: • Active • Passive • Interactive • Barriers: Time, volume, access, readability, relevance, motivation and org culture • A sense of being overwhelmed

24. Utility of the map • Overview of body of literature • Responsive to different knowledge needs • Concise; a starting point • Easy to update and expand • New insights into the nature of the gaps eg policy debates that highlight need for research; emerging practice that requires evaluations

25. Conclusions • Time intensive processes not feasible • Importance of secondary analysis to support accessibility • Mapping offers potential to make explicit strengths and limitations of the knowledge bases; an accountable way to manage the uncertainty • Potential of maps to highlight gaps in primary research and areas of substantial knowledge for in-depth reviews.

26. Implications • Policy and practice: • Need to go beyond established knowledge; • Basis for reasonable hypotheses • Further research – primary and secondary analysis • Systematic mapping methodology: • Boundaries of maps – content and types of knowledge • User involvement

27. Overall conclusions • To move beyond the rhetoric of being evidence- informed need to address gaps in both primary and secondary research • The need to engage users to identify questions and priorities