Developing revalidation for UK physicians

1. Developing revalidation for UK physicians Maintenance of competence in the age of technology Prof Dame Lesley Southgate: St Georges Medical School, University of London

2. Prof Dame Lesley Southgate: St Georges Medical School, University of London

3. Prof Dame Lesley Southgate: St Georges Medical School, University of London

4. Purpose of this talk • To show that evidence for revalidation of physicians can be collected and judged within a systematic, transferable framework which could apply to all physicians at any stage of their career • To ask you, as I discuss some of the slides, to think over ways that storage and exchange of this evidence could be achieved without distorting or losing its meaning or creating false synergies Prof Dame Lesley Southgate: St Georges Medical School, University of London

5. Revalidation has a primary purpose • to demonstrate that doctors on the register continue to meet the standards that apply to their own discipline • Implies a basic standard that nearly all doctors should reach Prof Dame Lesley Southgate: St Georges Medical School, University of London

6. It can also be seen to have a number of secondary purposes • To promote continuous professional development among practitioners • To encourage improvement in the quality of care, team working, communications and appropriate behaviour of practitioners • Implies standards which allow progress towards excellence to be documented Prof Dame Lesley Southgate: St Georges Medical School, University of London

7. It can also be seen to have a number of secondary purposes • To identify doctors, as far as is practicable, for whom there are significant concerns about their fitness to practise • To reassure, as far as is possible, the public and individual patients that their doctors are up to date and fit to practise • Implies accuracy around the ‘cut point’ for the basic standard Prof Dame Lesley Southgate: St Georges Medical School, University of London

8. Competence versus Performance Competence is necessary, but not sufficient, for acceptable clinical performance and this concept is especially important when assessing the performance of doctors in difficulty where it is essential to determine if poor performance stems from a compromised knowledge and skill base or not- the difference between can’t perform, or won’t which may be related to a failure of professionalism or poor health or social circumstances Prof Dame Lesley Southgate: St Georges Medical School, University of London

9. GMC Good Medical Practice • provides the broad domains underpinning a potential assessment framework • All Royal Medical Royal Colleges have elaborated the guidance to define the characteristics of the excellent (and unacceptable) physician. • GMP underpins every aspect of UK medical practice from undergraduate through residency to continuing professional development • It underpins licensure and certification Prof Dame Lesley Southgate: St Georges Medical School, University of London

10. Good Medical Practice • Can be mapped to Canmeds and the ACGME competencies • We have a framework/classification for how we record information about a physician through the continuum of practice Prof Dame Lesley Southgate: St Georges Medical School, University of London

11. Domains of performancefrom Good Medical Practice • Knowledge, skills and performance • Maintain your professional performance • Apply knowledge and experience to practice • Keep clear, accurate and legible records • Safety and Quality • Put into effect systems to protect patients and improve care • Respond to risks to safety • Protect patients from any risk posed by your health Prof Dame Lesley Southgate: St Georges Medical School, University of London

12. Domains of performancefrom Good Medical Practice • Communication, partnership and team work • Communicate effectively • Work constructively with colleagues and delegate effectively • Establish and maintain partnerships with patients • Maintaining trust • Show respect for patients and colleagues • Treat patients and colleagues fairly and without discrimination • Act with honesty and integrity Prof Dame Lesley Southgate: St Georges Medical School, University of London

13. The domains of performance • together determine the territory within which • high level statements of values and principles (criteria/standards) are set out • which then need elaboration in order to map to actual clinical practice • to set levels of achievement (levels/standards in the other sense) • identify the best methods by which evidence of that achievement can be gathered and judged for a defined purpose Prof Dame Lesley Southgate: St Georges Medical School, University of London

14. Headings for a blueprint for recertification Prof Dame Lesley Southgate: St Georges Medical School, University of London

15. Criteria and standards • From these, meaningful sub-divisions or criteriacan be defined (high level statements that encompass values and principles), leaving the term “standard”as the word to be used to define the level of quality or excellence in relation to each criterion Prof Dame Lesley Southgate: St Georges Medical School, University of London

16. And Evidence • The selection of tools to gather evidencewill, however, also depend on the characteristics of the different methods, published research in the field of assessment, the likelihood of legal challenge to recommendations perceived by individuals as perverse, and the contribution each tool can make to the provision of evidence that samples the territory described within the domains, in the context of the doctor’s actual practice. Prof Dame Lesley Southgate: St Georges Medical School, University of London

17. The utility index • The design of any assessment system will also need to take into account its utility. The utility, or usefulness, of an assessment has been defined as a product of its reliability, validity, feasibility, acceptability and educational impact. Prof Dame Lesley Southgate: St Georges Medical School, University of London

18. A programme of workplace assessment should be • Specified (domains, criteria, standards) • Developmental • Based on the collection of evidence using an appropriate variety of methods • Triangulated • Quality assured • These principles should be applied to any emerging framework for recertification. Prof Dame Lesley Southgate: St Georges Medical School, University of London

19. The importance of context and actual practice in performance assessment • Setting defensible standards for performance assessment is difficult • Assessment methods are required that provide evidence about how doctors respond to real clinical encounters and situations, rather than simulations.  • As each encounter occurs in unique circumstances, assessors may have less data on which to base a decision about the level of performance, whether it be recorded numerically or as a qualitative judgment. • Most performance assessment methods are relatively new, particularly when used for making decisions that might restrict clinical practice.  Prof Dame Lesley Southgate: St Georges Medical School, University of London

20. Sources of evidence for recertification of (for example) family physicians will be considered in three sections. • Clinical governance data routinely available (in theory!) from an employer and collected as a result of the doctor’s clinical practice. • Workplace assessment of the individual practitioner • Tests of knowledge and skills Prof Dame Lesley Southgate: St Georges Medical School, University of London

21. Attribution • The concept of using performance data from sources other than the assessment of individual doctors is not new. • controversial • debates about confidentiality and consent, individual freedoms and clinical autonomy, • no single organization or employer is likely to possess sufficient data to set standards for performance-based measures that reflect the scope of an individual’s clinical practice.  • need a framework for the development of standard-setting procedures that use patient-outcomes-orientated measures of clinical performance, based on a collaborative approach between several health care stakeholders Prof Dame Lesley Southgate: St Georges Medical School, University of London

22. Clinical governance data • Early indications have suggested that routinely gathered data across nine areas related to practice might generate some useful insights into practice level performance • Areas 1-4. May be useful for individual assessment: but there are problems with attribution. • All to be accompanied by development of standardised documentation, a commentary and discussion within the appraisal meeting. Prof Dame Lesley Southgate: St Georges Medical School, University of London

23. PACT • Prescribing Analysis and Cost (PACT) Data. Data must be attributable to individuals and prescribing in the acute situation, and within defined therapeutic areas is a promising approach. Recommended for piloting Prof Dame Lesley Southgate: St Georges Medical School, University of London

24. Referral Information • ….. there is conflicting evidence about the usefulness of these data and referral decisions may be better judged by methods proposed later. Not recommended Prof Dame Lesley Southgate: St Georges Medical School, University of London

25. Complaints • with a reflective piece to accompany quantitative data in the portfolio of evidence. The definition of a complaint, and it’s separation from informal concerns raised in the course of practice, will be essential. Recommended for piloting Prof Dame Lesley Southgate: St Georges Medical School, University of London

26. Significant Events • with specification for types of cases. Standard documentation and a commentary including own contribution to be part of the portfolio of evidence available for appraisal. Recommended for piloting Prof Dame Lesley Southgate: St Georges Medical School, University of London

27. Quality and Outcomes Framework (QOF) Data. • Area 5 Principally about the practice/team, but may be useful as a basis for reflection/discussion with appraiser (documentation to be developed) • These data will be difficult to use in their present form. Not recommended Prof Dame Lesley Southgate: St Georges Medical School, University of London

28. Practice achievement with the exception reporting tallies • The best candidate for recertification to be used as the basis for a written commentary in the portfolio of evidence available for discussion at appraisal • This is a major area for piloting, and must not be ignored because of political and academic difficulties Prof Dame Lesley Southgate: St Georges Medical School, University of London

29. Clinical governance data • we recommend use of the methods proposed above for piloting with volunteer PCTs. To include • development of standardised reporting documentation, • guidelines for how to present a commentary • training for appraisers • external review of judgements and recommendations in order to begin the definition of standards. Prof Dame Lesley Southgate: St Georges Medical School, University of London

30. Workplace assessment of the individual practitioner • The range of methods proposed below is included because together they will provide evidence in the domains for recertification/revalidation, to be considered, triangulated and judged with data generated from clinical governance activities. Prof Dame Lesley Southgate: St Georges Medical School, University of London

31. Medical record review • We recommend piloting in conjunction with volunteer PCTs and practices. • The work to include • survey of present approaches to examining records during PCT visits to practices • development of standardised documentation • training of new, and identification of experienced reviewers • and standard setting and judgments Prof Dame Lesley Southgate: St Georges Medical School, University of London

32. Case based discussion (CbD) • We recommendthis tool for recertification as part of accredited CPD and with one formal event as part of the 5 year appraisal cycle. Piloting as part of a prototype will be necessary. Prof Dame Lesley Southgate: St Georges Medical School, University of London

33. Multi-source feedback • The choice of tools will be based on their evaluation against the utility index. MSF cannot stand alone to provide evidence for recertification. It should be triangulated with other evidence in order to stand legal challenge. Data protection, freedom of information and attention to the provision of feedback are crucial, as is central data management. Prof Dame Lesley Southgate: St Georges Medical School, University of London

34. Patient satisfaction questionnaires (PSQ • The review by the Picker Institute Europe 17 has identified five patient feedback questionnaires that have been used within the UK and might be considered (after suitable modification) for use within a framework of recertification. Prof Dame Lesley Southgate: St Georges Medical School, University of London

35. Formal Competence testing methods • The possibility of using the certifying examination/methods of assessment on a voluntary basis to satisfy some of the evidential requirements as part of CPD or as evidence to include in their recertification portfolio. This will not be a requirement and will be available only on request. Further discussion will be necessary to determine the weight that might be placed on these tests for doctors who pass them, and what other evidence they may substitute. Prof Dame Lesley Southgate: St Georges Medical School, University of London

36. Managed CPD:1 • Will be based on GP centred reflection and learning. Linked to “Essential General Practice,” this will be a systematic approach to CPD requiring GPs to demonstrate assimilation of new and evolving knowledge through an e-learning format. Prof Dame Lesley Southgate: St Georges Medical School, University of London

37. Managed CPD:2 • Some of the tools for WBA will form part of CPD within a programme where the assessor and the doctor are peers, both of whom are participating in an accredited programme to be recorded in the portfolio. As confidence grows in the process formal occasions may be requested to provide evidence of the doctor’s knowledge base. • We recommendthat this approach be piloted soon, as we are certain that a minimum data set that depends heavily on MSF and untried clinical governance data will be a high risk strategy Prof Dame Lesley Southgate: St Georges Medical School, University of London

38. Synthesis Prof Dame Lesley Southgate: St Georges Medical School, University of London

39. Post script: who will judge the evidence? • The training needs of appraisers are not addressed in this presentation. They are considerable, and in particular they will be acute in the arena of feedback. The potential to do harm is greatest in this area, either to individuals or teams. But the appraiser is not the judge and jury for recertification, or indeed revalidation. While they will make judgements, they will be one of many, made both within the evidence assembled for the portfolio, and by other appraisers within the 5 year cycle Prof Dame Lesley Southgate: St Georges Medical School, University of London

40. Prof Dame Lesley Southgate: St Georges Medical School, University of London