NO CONFLICT OF INTEREST TO DECLARE

1. NO CONFLICT OF INTEREST TO DECLARE

2. Can peer review improve quality? David Smith

3. Peer review origin? Ethics of the Physician Ishaq bin Ali al-Rahwi (854–931) of Syria

4. All Quality Assurance programmes seem to rely ultimately on peer review, often in the form of an expert panel, to set the standards and the limits of acceptability.

5. Problems with peer review • Provision of • objective standards • structured processes • training reviewers • Perverse incentives/perverse activities • Reviewer bias • Authority of the reviewers • Indemnity/protection

6. GMC procedures to assess the performance of doctors who may be seriously deficient • peer review of the doctor’s practice at the workplace • tests of competence and skills. Peer reviews are conducted by three trained assessors, two from the same speciality as the doctor being assessed, with one lay assessor and include: • the doctor’s portfolio of training, experience, practice and self assessment • a review of the doctor’s medical records • discussion of cases selected from these records • observation of consultations • a tour of the doctor’s workplace • interviews with at least 12 third parties (five nominated by the doctor) • structured interviews with the doctor.

7. Failure of peer review Given the magnitude of the resources devoted to quality assurance and the centrality of peer assessment to these efforts, the results of this literature analysis indicate the need for a global reexamination of the peer review process. Several of the proposals discussed herein appear to have considerable potential for improving the reliability of peer judgments. Research directed at evaluating this potential should be part of the overall reassessment of peer review. Goldman RL JAMA. 1992;267:958-960

8. Reviewer agreement physician agreement regarding quality of care is only slightly better than the level expected by chance. Goldman RL JAMA. 1992;267:958-960

9. Goldman suggestions • Use of multiple reviewers • More objective assessments • Higher standards for peer reviewers • Use of practice guidelines • Use of outcome assessments

10. Watching the doctor watchers The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through • improved reviewer selection and training • a structured review method • more physician reviewers per record Rubin et al JAMA 1992;267:2349-2354

11. Review of 339 hospitals PR system • Only 24% of respondents said it was very likely to improve quality while 33% said it was unlikely • Needs standardisation, • more governance, • bigger case load for reviews (>1% of admissions), • more personnel (already 1.1wte per 100 beds) The Process of Peer Review in U.S. Hospitals Marc T. Edwards & Evan M. Benjamin JCOM 2009 Vol. 16;10 461-467

12. Reviewer agreement • 13 board-certified physicians completed a total of 741 structured implicit record reviews of 95 records for patients who experienced severe in hospital adverse events while in the hospital (hypokalemia, hyperkalemia, renal failure, hyponatremia, and digoxin toxicity). • They independently assessed the degree to which each adverse event was caused by medical care and the quality of the care leading up to the adverse event. • Working in pairs, they then discussed differences of opinion, clarified factual discrepancies, and re-rated the record. The authors compared the reliability of each measure before and after discussion, and between and within pairs of reviewers. • The reliability of the assessment of whether the complication was iatrogenic was 0.46 before discussion and 0.71 after discussion between paired reviewers, indicating considerably improved agreement between the members of a pair. • However, across reviewer pairs, the reviewer reliability was 0.36 before discussion and 0.40 after discussion. • Similarly, for the rating of overall quality of care, reliability of physician review went from 0.35 before discussion to 0.58 after discussion as assessed by pair. However, across pairs the reliability increased only from 0.14 to 0.17 Conclusions. When 2 physicians discuss a record that they are reviewing, it substantially improves the agreement between those 2 physicians. However, this improvement is illusory, as discussion does not improve the overall reliability as assessed by examining the reliability between physicians who were part of different discussions. Hoper TP et al Medical Care 2000, 38:2 152-161

13. Guidelines Surrogate markers of quality May be ill founded May be biased Open to interpretation Unintended consequences Limit development

14. Are guidelines following guidelines?A review of 279 guidelines 1985-1997 Guidelines published in the peer-reviewed medical literature during the past decade do not adhere well to established methodological standards. While all areas of guideline development need improvement, greatest improvement is needed in the identification, evaluation, and synthesis of the scientific evidence. JAMA. 1999;281(20):1900-1905

15. Practice guidelines developed by specialty societies: the need for a critical appraisal 431 guidelines were eligible for the study. Most did not meet the criteria Despite improvement over time, the quality of practice guidelines developed by specialty societies is unsatisfactory. Explicit methodological criteria for the production of guidelines shared among public agencies, scientific societies, and patients' associations need to be set up. Lancet 2000:355;103-106

16. Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines Recommendations issued in current ACC/AHA clinical practice guidelines are largely developed from lower levels of evidence or expert opinion. The proportion of recommendations for which there is no conclusive evidence is also growing. These findings highlight the need to improve the process of writing guidelines and to expand the evidence base from which clinical practice guidelines are derived JAMA. 2009;301(8):831-841

17. Risk adjusted outcomesfunnel plots

18. BCIS ‘peer review’ and angioplasty quality Historical ‘no holds barred’ case review is an exemplar of peer review but to be rigorous it needs a systematic format and a written record

19. A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow? C J A Bowles GUT 2004 53 277-283

21. Can peer review improve quality? Increasingly robust risk adjusted outcome data Local systematic egalitarian case review arrangements

22. BCIS Peer Review • What about BCIS per review • BCIS were asked to set up a PR system in 1999 and felt individual PR was not feasible and designed a unit PR system.The first draft had many of the features that Goldman suggests but fell at the first few hurdles of reviewer acceptability, resources, lack of indemnity • At that point it was highjacked by the department of Health in order to limit the potential proliferation of of low grade angioplasty units. It has continued in this format of a box ticking exercise for new units measured against current BCIS guidelines. • limited as it is does it improve quality? One of the few but always contentious, surrogate measures of quality that are recorded is the number of procedures done per year. • Yet there are some units consistently doing less than 400 for up to 7 years

23. Numbers of small volume units over last ten years

25. 2006 data: Ludman Number of PCIs performed in 2006 Centres performing < 200 procedures PCI program start date

26. The purpose of peer review is to provide an expert view of an individual’s ability to meet professionally acceptable standards and thus help ensure an acceptable level of continued professional competence. Such review might necessarily identify individuals who do not conform to acceptable standards and who may therefore be required to undergo retraining or be suspended. If the outcome of a peer review might put at risk the ability of an individual to continue to practice, then such a review has to be rigorous in the extreme. It would need to be supported by a large volume of accurate statistics and involve comparison to a detailed set of performance standards that is accepted by the speciality as a whole

27. Audit • Case review • Direct observation of clinical activity/procedures • 360 Appraisal