Measuring the Important Regular Practice Review

1. Measuring the ImportantRegular Practice Review IPAC / CPE Conference 5 October 2012 Dr John Adams - Chairperson David Dunbar – Registrar Medical Council of New Zealand

2. Presentation Outline • The New Zealand environment • legal framework for recertification in New Zealand • Regular practice review • philosophy and background • targeted approach • tools • Challenges encountered with implementation of RPR • Next steps

4. New Zealand – the numbers Population 4.4 million (NZ) Ireland 4.6 million Sydney 4.5 million Washington 4.6 million Toronto 5.6 million Land area (sq kilometres) 270,000 (NZ) Alberta 256,000 Victoria (Australia) 227,000 United Kingdom 244,000 Colorado 270,000

5. Doctors in New Zealand – a profile • Nearly 14,500 registered doctors • 1,550 new doctors registered each year • Around 40% of doctors have overseas primary medical qualifications, coming from over 100 countries • Principal categories of registration (scopes of practice) • 9,432 Vocational (35 scopes of practice) • 7,889 General • 3,510 Provisional General

6. Legal framework in New Zealand The Health Practitioners Competence Assurance Act 2003 regulates 22 professions and over 70,000 health practitioners, through 16 regulatory authorities The Act’s purpose: to protect the health and safety of members of the public by providing for mechanisms to ensure that health practitioners are competent and fit to practise their professions MCNZ ‘s legal functions include to review and promote the competence of doctors, and to recognise, accredit, and set programmes to ensure ongoing competence

7. Trust and confidence • NZ Doctors are respected for the high standard of care they provide • Most doctors meet high standards routinely and are dedicated to lifelong learning • Medical regulation needs to support this professionalism • Regulation should be as much about improving and assuring the standards of doctors, as it is about identifying and addressing poor performance • MCNZ’s overriding interest in the safety of patients and the quality of care they receive from their doctor • Medical regulation must provide a framework to maintain trust and confidence

8. Recertification as one mechanism to assure competence • CPD has been mandatory for doctors since 1996 • MCNZ historically relied on CPD as a principal mechanism for assuring doctors’ competence for recertification • Large majority of doctors responsibly meet CPD requirement • Now we recognise more focus needs to be placed on the effectiveness of CPD activities on performance

9. General scope recertification requirements • A professional development plan • 20 hours of continuing medical education • 10 hours of peer review • Annual participation in audit of medical practice • Minimum number of meetings with nominated collegial relationship provider (6 the first year, 4 yearly thereafter) • Essentials test in the first year of enrolment and then every 3 years • Multisource or patient feedback questionnaires (every 3 years) • Regular practice review (also every 3 years) • www.inpractice.org.nz

10. Regular Practice Review • Workplace based assessment can vary from longer term observation and the use of assessment tools in the doctor’s everyday work, through extended courses or assessment processes, to practice visits of a day or two’s duration. • In our conceptualisation, RPR is a workplace based practice visit using a variety of tools and observations over one to two days. • A formative process as a part of CPD, similar in format to our competence assessments, but with the primary purpose of improving doctors’ practice.

11. Purpose of regular practice review • To help maintain and improve standards of the profession – a quality improvement process • May also assist in the identification of poor performance GOAL • To help individual doctors identify areas where aspects of their performance could be improved, benefiting not only their own professional development but also the quality of care that their patients receive.

12. MCNZ initial targeted approach • General scope - RPR is now mandatory for doctors registered in a general scope of practice (as one component of a structured and strengthened recertification programme) • Vocational scope – doctors may choose to participate in RPR on a voluntary basis. MCNZ is working closely with medical colleges encouraging them to develop RPR processes and make them available to doctors as a component of CPD activities.

13. RPR Context • Our view is that RPR should be primarily based in the profession and that our role as a regulator is to support and encourage this. The programme for General Registrants is a more active first step with the regulator taking a very clear lead. “Professional self-regulation is dependent on effective and credible assessment. From this perspective, assessment becomes an important act of professionalism by the physician. (6,10)” “assessment should not be just about identifying the incompetent physicians.(13) Holmboe E, Journal of Continuing Education In The Health Professions, 28(s1):s4–s10, 2008

14. Workplace-Based Assessment – why bother? “Although many forms of assessment can be used to show a doctor’s knowledge or competence, there is evidence that competence does not reliably predict performance in clinical practice (6); one major advantage of workplace based assessment is its ability to evaluate performance in context. (7)” Miller A, Archer J. BMJ 2010;341:c5064 “…doctors’ abilities when assessed in a controlled (deconstructed) environment do not dependably predict their actual day-to-day performance.(7,8)” Crossley J, Jolly B. Medical Education 2012: 46: 28–37 “Doctors have limited ability to self-assess their learning needs…… The worst accuracy is in the least skilled and the most confident.” JAMA 2006;296:1094-1102

15. What activities do not show improved performance or outcomes? • Formal CME conferences without enabling or practice-reinforcing strategies have little impact • Didactic sessions • Self assessment and self directed learning • Large group teaching

16. What activities do improve performance? • Interactive programmes • Providing doctors with their own performance data • Activities based on identified learning needs. • Activities anchored in a doctor’s own work environment and individual practice • assessed by an external body or person, rather than based on self evaluation

17. Tools commonly used • Observation of clinical activities • Records review • Discussion of clinical cases • Chart stimulated recall • Case based discussion • Multisource feedback Miller A, Archer J. BMJ2010;341:c5064

18. RPR Process In a 2012 review, Crossley and Jolly make a convincing case for subjective global assessments by experienced observers in assessing performance. “In essence, scraping up the myriad evidential minutiae of the subcomponents of the task does not give as good a picture as standing back and considering the whole.” Crossley J & Jolly B, Medical Education 2012: 46: 28–37

19. RPR Process “Four general principles emerge: the response scale should be aligned to the reality map of the judges; judgements rather than objective observations should be sought; the assessment should focus on competencies that are central to the activity observed, and the assessors who are best placed to judge performance should be asked to participate.” Crossley J & Jolly B, Medical Education 2012: 46: 28–37

20. Is there evidence that ‘RPR’ works? “Considering the emphasis placed on workplace based assessment as a method of formative performance assessment, there are few published articles exploring its impact on doctors’ education and performance.” Miller A, Archer J. BMJ 2010;341:c5064

21. Overeem K et al, Medical Education 2007: 41: 1039–1049

22. Workplace-Based Assessment Impact “Eight studies examined multisource feedback,(17-24) four concentrated on the mini-clinical evaluation exercise,(25-28) one investigated direct observation of procedural skills,(29) and three looked at multiple assessment methods.(30-32)” “There is now convincing evidence that systematic feedback delivered by a credible source can change clinical performance,” “The strongest evidence for workplace based assessment improving performance comes from studies examining multisource feedback.” “We were unable to unearth any clear evidence to show that the mini-clinical evaluation exercise, direct observation of procedural skills, or case based discussion can lead to improvements in performance.” Miller A, Archer J. BMJ 2010;341:c5064

23. Challenges in Implementation • Doctor resistance • “Is there any evidence that it works?” • Stakeholder concerns: • Royal Colleges and Specialist Societies: • Bi-national differences • Evidence • Placing requirements on their members • Capacity • Cost • District Health Boards and other employers: • Cost • Capacity • Service efficiency • Unions • Will this be used by employers in disputes with employees? • Implications for an already stretched workforce.

24. Challenges in Implementation • The MCNZ has consulted on the use of RPR widely. • In discussion with Colleges and Societies the MCNZ has been clear that it sees RPR as a very useful addition to CPD requirements. • We have been clear that we are not at this stage requiring RPR in CPD regimes for Vocationally Registered doctors, but that we support its development. • Some Colleges and Societies (O&G, Orthopaedics) have taken this up. • We have required RPR as a part of our new CPD Programme for General Registrants.

25. NZ Orthopaedic Association pilot report

26. NZ Orthopaedic Association pilot report

27. NZ Orthopaedic Association Pilot

28. Next steps • Consideration of a risk based screening approach, that is identifying risk factors, and targeting mandatory RPR to those who demonstrate particular risk • Continue to work closely with medical colleges encouraging implementation of RPR • Evaluation of effectiveness (for example longitudinal study) of RPR

29. Regular Practice Review Discussion

30. IPAC 2013 Queenstown, New Zealand Monday 30 September – Wednesday 2 October 2013