Generating evidence from practice by defining and measuring engagement outcomes in

1. Generating evidence from practice by defining and measuring engagement outcomes in Family Services work. Julie Boffa, Project Manager, NEMC&FSA John Cheshire, Senior Manager, Community & Family Services, Brotherhood of St Laurence, Executive representative, NEMC&FSA ACWA 2010

2. This presentation will… • Describe work undertaken within the North East Metro Child and Family Services Alliance to define and measure families engagement with family support services • Position this work within wider aspirations to generate ‘practice based evidence’ through routine outcome measurement and feedback

3. Who is the NEMC&FSA? North East Metro Child & Family Services Alliance • 9 community service agencies, in partnership with Child Protection, bound by MOU • Formed in 2007 to provide Integrated Family Services for children at risk of report to Child protection, or with substantial difficulties • Supported by central intake, Child FIRST • Services 5 LGAs in North East Melbourne - population the size of Tasmania; highest metro Aboriginal pop; many indicators of disadvantage.

4. Our clients in brief • For 2008-09, 89% of families referred to Child FIRST had one or more complex characteristic: • Child Protection involvement • Mental illness • Substance abuse • Family violence • Physical or intellectual disability • Sexual assault • Juvenile Justice involvement • 46% had known prior or current Child Protection involvement at the point if intake • Children may demonstrate a range or emotional and behavioral issues

5. What is ‘practice based evidence’? Practice-based evidence has two elements: • gathering evidence of the effectiveness of one’s practice or service as it is delivered • on the basis of this evidence, modifying one’s practice to achieve better results(Moore, 2010The nature and role of practice based evidence, Building practice based evidence through the evaluation of parenting and early childhood intervention programs, ARACY Collaborative Workshop, Melbourne) • Compliments evidence based practice to close the ‘evidence informing circuit’ • Evidence in, evidence out, evidence in…

6. Measuring engagement outcomes • In this context, NEMC&FSA sought to better understand service engagement outcomes in order to be more responsive to families and improve service outcomes • A component of the NEMC&FSA Ian Potter Foundation funded project, in partnership with the Parenting Research Centre, to bring evidence to practice with intensive need, vulnerable families • First step was to undertake an audit of engagement outcomes

7. NE Engagement Audit • A practice research project with team leaders, undertaken by NEMC&FSA Project Manager, supported by final year social work student (Gemmah Melia) • Included 40 cases requiring allocation in March 2009 • Sample was typical of NE allocations over 2009-09: • 70% referred via Child FIRST • 48% current or prior CP involvement • 85% significant wellbeing concerns or complex issues (eg. MH; FV; D&A; CP) • Follow-up interviews with workers took place over September/October 2009

8. Defining engagement in Lit. •  Refers to early phases of work with clients • Joining (Gitterman 1996) • Forming a relationship (Krill 1996) • Establishing goals (Reid 1996) • Forming a relationship (Saleeby 1997) • Both a cause of participation and its result, a causal influence and an outcome:“An umbrella term that legitimately includes behaviours (eg. service usage, duration and completion) as well as attitudinal or affective dimensions” (Yatchmenoff 2005)Literature review provided to NE Engagement Audit by Parenting Research Centre

9. Defining engagement in Audit Cases were thematically coded to establish engagement descriptors within three categories: • Engaged identified in vignettes as including both: • A constructive relationship or attitude component • A constructive activity component Like literature, includes both attitude & behaviours • Did not engage - absence of both these criteria • Indeterminate - Presence of only one criterion, or performance too weak or variable to satisfy one or both criteria

10. Descriptors: Engaged • Open to services Family cooperates, motivated & follows through with intervention • Community & service links, short Referral and linkage focus, case closed when initial goals met • Warmed to support Family initially reluctant or wary but becomes open to support with time • Persistent effort* Family overall willing to work with services but constant crisis or difficulties; and/or feeling family engagement dependent on worker effort • Child &/0r YP focused Focus is child or young person, including where parent reluctant to become involved • Established but disrupted Case closed after constructive progress due to external reason eg worker leaves; report to Child Protection; family moved after engagement; OR family ceases engagement/withdraws after initial progress *Added after audit as identified through routine follow-up

11. Descriptors: Did not engage • Declined service, closed to offers - Family actively asserts lack of interest • Unavailable - Family indirectly conveys lack of interest eg misses appointments (no show or continual excuses); no or intermittent response to letters or phone calls

12. Descriptors: Indeterminate • Family identified other service needs Family involved with or states preference for other legitimate service need (may cross over with unavailable) • Open to support, no identified FS role FS decision that FS role would not benefit family or be the most appropriate service response • Family moved - Before intervention established • Patchy engagement On-off pattern of contact or follow through; repeated swings between unavailable and open; multiple disruptions; meets only one engagement criteria or weak on both

13. Routine follow-up • Using the engagement Types and Descriptors generated through the initial NE engagement audit, from January 2010 NEMC&FSA now routinely follows up engagement outcomes six months from the month of allocation for all allocated cases • Administered & collated by NEMC&FSA Project Manager based on feedback from Team Leaders at the 9 agencies

14. Findings after first 6 months

15. Fluctuating outcomes • Outcomes to date have fluctuated between the audit and quarterly samples • More analysis is required to determine any underlying influences and/or more time is needed for trends or patterns to become evident • Influences may include: • Improved performance • Variability between different raters • Differences in samples demographics, eg: • Different client characteristics – for instance reduced Child Protection referrals: Audit 30%; Jan-Mar 16%; Apr-June 17% (impacted by demand management action June to December 2009 when cases allocated)

16. Mediating influences • The initial Engagement Audit found poorer outcomes for: • Child Protection referrals • Prior Family Services involvement • Significant wellbeing concerns cases compared to ‘Complex IRIS issues’ with best outcomes for ‘Other IRIS issues’ ie reflects presentation severity gradient • Generally, the first quarterly samples have found better outcomes in all categories, leading to the improved engagement results reported • No effect was found for length of time families received ‘active holding’ prior to allocation to an ongoing worker

17. Does engagement mean change? • Requires measure of family progress to answer • Next challenge for ‘practice based evidence’ quest! • To explore, NE Engagement Audit trialed use of ‘Strengths and Stressors tracking tool’ (Berry PhD, 2008) • Adapted tool by substituting headings with Victorian ‘Best Interests’ domains & converted Berry’s ‘-3 to +2’ scale to ‘1 – 6’, based on worker feedback • 1 greatest stress, 6 greatest strength, & 4 ‘community norm’ (‘0’ on Berry’s scale) • At the time of engagement follow up interviews, workers rated their view of family strengths or stressors at allocation & as current or at closure

18. Tracking progress using theStrengths and Stressors • Found to be a useful indicator of change - though this application very subjective • Average pre-ratings on all dimensions were less than ‘4’ (=community norm/more strength than stress) • All dimensions recorded increase in average ratings over intervention based on worker ratings • Highest av. post score & equal greatest increase was for Community Partnerships & Social Networks • Equal high change for Child Safety, then Parent Capability; Social & Economic Environment • While still positive, lowest change: Child/Young Person Development, Well Being & Stability

19. Cautions • This work is new and there is a lack of comparative data against which to benchmark expectations about outcomes – in a context of hard to reach, complex family presentations • For instance, a conclusion of US studies of child abuse and neglect prevention programs was:“Despite extended efforts and a clear, strength-based approach to service delivery, the majority of families reached by prevention programs will leave before reaching their service goals or achieving the service levels articulated in the program’s model” (Daro and Cohn Donnelly (2002) p.737).

20. For more information Contact:Julie Boffa Project Manger North East Metro Child and Family Services AllianceLocated at Children’s Protection SocietyVictoria jboffa@cps.org.au