Outcome Measurement

1. Outcome Measurement Train the Trainer Program Day 1

2. Day 1 Background and Rationale The task of Local Trainers Data Collection Protocol Wellbeing Module Managing Change Day 2 Overview Additional Data items Clinician-rated measures Consumer Self Report measure Delivering Training Outline of Training Program

3. Day 1 Workshop Objectives At the completion of this session participants will be able to: • Describe the National Mental Health Strategy and the need to ensure the quality and effectiveness of service delivery through outcome measurement • Identify which outcome measures are to be used • Identify the points during service delivery at which data must be collected • Discuss the challenges and opportunities of outcome measurement in their services

4. We will cover • How we got here and why - National Mental Health Strategy context • What are the expected benefits? • Challenges for implementation and current status • National infrastructure, including what happens to the data

5. A logical ‘next step’ ... • Widespread movement to an outcomes orientation across the health system • National Mental Health Policy, 1992 “To institute regular review of client outcomes of services provided to persons with serious mental health problems and mental disorders as a central component of mental health service delivery” • Research and development 1993-98

6. What wasn’t achieved in the first 5 years “Information in mental health is grossly undeveloped. The lack of nationally comparable data on service outputs, costs, quality and outcomes places major limitations on the extent to which the National Mental Health Strategy can achieve its objectives.” Evaluation of the National Mental Health Strategy, Dec 1997

7. The place of information in the National Mental Health Strategy1993 - 1998 • Lack of quality information hampered mental health services reform • Mental health poorly represented in national health information collections • The National Strategy gave priority to improving quality and quantity of mental health information

8. The 2nd National Mental Health Plan1998-2003 Three major themes : • Service Quality and Effectiveness • Prevention and Promotion • Partnerships in Service Reform and Delivery

9. The guiding question ... • Who receives ..... • What services ..... • From whom ..... • At what cost ...... • With what effect ...… from Leginski et al 1989

10. Data is collected for two purposes • Two requirements need to be met: • Outcome Measurement requires recurrent snapshots of the consumer’s clinical status, at least at the beginning and end of the episode • Casemix needs an overall judgement about the consumer’s needs for care during the episode, usually based on an assessment at one point in time (eg HoNOS at admission, PDx at discharge) • Most measures are used for both purposes

11. Why? • Broadly, the aspiration is to build an informed mental health system where information is available to guide decisions at all levels to: • support clinicians in their treatment decisions • inform consumers about the services they receive • help managers manage • inform policy makers in planning and paying for services

12. OM at the Service Level • Offers a systematic means of documenting progress – can highlight unexpected progress, deterioration or maintenance • Assists case reflection • Facilitates dialogue with consumers • Standardises assessment tools/processes • Provides feedback re impact of interventions, procedures, policies, training • Support management decisions (eg case allocation?)

13. Outcomes and Casemix Data can help with • Quality assurance activities • Service utilisation reviews • Interpretation of consumer outcomes • Development of clinical protocols • Justification of service need

14. A brief history of Outcomes Research & Development1993-2002 • 1994 Uni of New South Wales - reviewed available outcome measures, recommended 6 to be trialed • 1996 Uni of Queensland - trialed the measures • 1998 Review of options for child & adolescent outcome measures (Bickman et al) • 1996-98 Mental health casemix development (MH-CASC Project) • 2000 – 2002 Round One implementation in four Victorian agencies. Data available and analysed

15. Key Features of Implementation • Outcome measures to be collected by all public mental health services (private hospitals too!) • Few international precedents (but lot of talk about it) • Measures to include ratings made by clinicians and consumers • All clinical staff to undergo training • Clinical leadership will help “sell” the message! • National approach to analysis and benchmarking

16. The Victorian Approach • Designated contact officer for each AMHS • Local information sessions • Best Practice Forums • OM Website • Enhancements to RAPID/CMI - Wellbeing module & Wellbeing Reporting Tool • Local Implementation Plans • Financial support

17. Tasks of Local Trainer • To provide local rater training • Check with your service but you may also have a role in: • Acting as an advocate for OM • Acting as resource for protocol clarification • Supporting measures completion • Troubleshooting data collection

18. What help is available • Train-The-Trainer course • Training Materials (manual, vignettes) • Website covers all aspects of OM • DHS OM team • Local Implementation group and Outcomes Coordinator • Helpline on Wellbeing Reporting Tool • Subject to availability, experts to support you in delivering training

19. Training Materials • Training Manual which includes • Background and Rationale • Protocol information and rating guidelines • Age-specific inserts with: • Training Vignettes and recommended ratings • Sample rating sheets for training • Video vignettes for adult MHS • CD-ROM • Clinicians’ Reference Guide

20. Criteria for Instrument Selection (1) • Applicable to Mental Health • collect information on symptoms, functioning, disability, quality of life • address all dimensions • suitable for consumers • useful for clinicians • data can be aggregated for managers • identical or similar measures across all settings

21. Criteria for Instrument Selection (2) • Acceptable • Minimum rater demand - brief • fit clinical process • Practical • minimal cost • scoring & interpretation simple • minimal training required • Valid • Reliable • Sensitive to change

22. The Suites

23. Clinician rated Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) Children's Global Assessment of Functioning Scale (CGAS) ICD-10 Factors Influencing Health Status Consumer self-report Strengths and Difficulties Questionnaire (SDQ) Outcomes and Casemix Measures for Children and Adolescents

24. Outcomes and Casemix Measures for Adult Mental Health Services • Clinician rated • Health of the Nation Outcome Scales (HoNOS) • Life Skills Profile (LSP-16) • Focus of Care (FOC) • Consumer self-report • BASIS-32

25. Outcomes and Casemix Measures for Older Persons • Clinician rated • Health of the Nation Outcome Scales (HoNOS 65+) • Life Skills Profile (LSP-16) • Resource Utilisation Groups - Activities of Daily Living (RUG-ADL) • Focus of Care (F)C) • Consumer self-report • BASIS32

26. What does the data collection protocol describe? • The NOCCData Collection Protocol defines: • What data is to be collected • When that data is to be collected • Train to revised Victorian version of Protocol - SDQ added, other changes minor • Who should take responsibility for collecting the data should be determined at a local level

27. Requirements to be met by the protocol at all levels • The data collection protocol should be clinically meaningful -- it should be consistent with and encourage good clinical practice • The data collection protocol should not be overly complicated • The protocol must be give rise to data that can be statistically analysed • The protocol should assist individual services to collect data at the most appropriate occasions that are consistent with generally agreed criteria

28. The Key Concepts • Episode of Mental Health Care • Mental Health Service Setting • Collection Occasion • Age Group

29. Mental Health Service Setting • 3 service settings = 3 ‘episode types’ • Inpatient episodes (Overnight admitted) • Community Residential episodes (24 hour staffed) • Ambulatory episodes • See note b. under the data dictionary definition of Mental Health Service Setting • Includes any hospital-based services for consumers who are not in overnight inpatient care • A specific data collection protocol is defined for each Mental Health Service Setting

30. Age Group • Which clinical measures are to be reported depend on the broad Age Group to which the consumer is assigned • Three Age Groups • Child & Adolescent (less than 18 years) • Adult (18 to 64 years inclusive) • Older people (65 years and older) • Service unit determines which measures are applied

31. Episode of Mental Health Care • Defined as “a more or less continuous period of contact between a consumer and a Mental Health Service Organisation that occurs within the one Mental Health Service Setting” • Two business rules: • ‘One episode at a time’ • ‘Change of setting = new episode’ • Start and end of each episode triggers a new collection

32. Collection Occasion • 3 Collection occasions • Admission = start of an episode of mental health care • Discharge = end of an episode of mental health care • Review = where an episode lasts for more than 91 days • The Collection Occasion is the unit of reporting – all data “belongs” to a Collection occasion • Within each Mental Health Service Setting, a specific dataset is required at each Collection Occasion ….

33. Course of Illness Episode of Care I Setting = Inpatient Episode of Care II Setting = Community Episode of Care III Setting = Community Ward A Ward B MST CATT MST Continuing care . . . 91 days Intake to community Discharge from community Discharge from inpatient 91 day review Admission to inpatient Episodes, cases, and collection occasions Case 1 Case 2 Period of no care Intake to community Collection occasions

34. Defining the individual Collection Occasion • Mental health service setting • Age group • Collection occasion date The reference date for all data collected at any given Collection Occasion (not the date individual measures were completed). • Reason for collection (for RAPID data entry) 01 Admission to inpatient 02 Discharge from inpatient 03 Admission to residential 04 Discharge from residential 05 Intake to community 06 Discharge from Community 07 ISP Review 08 Review (other)

35. Reviews • Reviews are required 91 days after a change in service setting and at 91 day intervals from the start of the episode • This may be different to the ISP review date, which is currently required at 6 month intervals after the opening of a case regardless of service setting • Align the required outcome collection as much as possible with the ISP (select ISP Review) • Those required 91 day review outcome measures not completed as part of ISP (select Review Other for data entry)

36. The NOCC Data Collection Protocol in Victoria

37. Specific issues in the implementation of the Data Collection Protocol • Movement between inpatient and community settings • Transfer between two wards of a psychiatric unit • Transfer between psychiatric units in different hospitals • Transfer of care between community teams • Multiple team involvement in case management • Intended same day admissions • Discharge from hospital on indefinite leave • Return to hospital from indefinite leave • Brief inpatient episodes • Rapid readmission to hospital • Assessment only cases seen by community teams • Consumers seen regularly, but at less than 3 monthly intervals

38. 1. Movement between community and inpatient settings Do ratings need to be recorded for the end of the community episode, as well as at the beginning of the inpatient episode, when a consumer is transferred from ambulatory care to overnight inpatient care ?YES

39. 2. Transfer between two wards of a psychiatric unit Is the transfer of a patient from one psychiatric ward to another, within the same hospital, a new episode requiring new data collection ?NO

40. 3. Transfer between psychiatric units in different hospitals Should a new inpatient episode be commenced when a consumer is transferred from one hospital to another, within the same mental health care organisation ?YES

41. 4. Transfer of care between community teams Does a new cycle of data collection begin when case management is transferred from one ambulatory care team to another within the same organisation ?NO

42. 5. Multiple team involvement in case management Is each team expected to complete ratings on the consumer ?NO

43. 6. Intended same day admissions Is each day of care a new inpatient episode, requiring a full set of ratings ? NO

44. 7. Discharge from hospital on indefinite leave Does an inpatient episode of mental health care continue when a patient is placed on extended leave but remains, legally, an inpatient ?NO

45. 8. Return to hospital from indefinite leave Does a new inpatient episode begin when a patient returns to hospital after a period of extended leave ?YES

46. 9. Brief inpatient episodes Are discharge ratings required for very brief inpatient episodes? for episodes of less than 3 days, NO otherwise, YES

47. 10. Rapid readmission to hospital If a patient is discharged from an inpatient unit and is readmitted within a very short period, is this a new inpatient episode or a continuation of the previous one ? It is a NEW EPISODE

48. 11. Assessment only cases seen by community teams Is outcomes and casemix data required on every person seen by community teams, regardless of whether they are accepted for treatment ?NO

49. 12. Consumers seen regularly, but at less than three monthly intervals How should the 3 monthly (91 day) review ‘rule’ be applied in these cases ? Does it mean that they will need to be seen more regularly ?NO

50. Managing Change