Quality Indicators in Gynecologic Cytology: Prospective and Retrospective Review

1. Quality Indicators in Gynecologic Cytology: Prospective and Retrospective Review Dr. Jennifer Brainard Section Head, Cytopathology Pathology and Laboratory Medicine Cleveland Clinic

2. Gynecologic Cytology • CLIA ‘88 mandated quality metrics for gynecologic cytology laboratories • Gynecologic cytology practice has changed dramatically • Liquid based Pap tests • Automated screening • High risk HPV testing • Proficiency testing • New screening guidelines

3. Gynecologic Cytology • New methods of monitoring performance are necessary • Groups from major societies convened to update quality indicators to reflect current practice • CAP Gynecologic Cytopathology Quality Consensus Conference • ASC Productivity and Quality Assurance in the Era of Automated Screening Task Force

4. CAP Consensus Conference • CAP was awarded a cooperative agreement by the CDC Goal: Identify what quality metrics are collected, how metrics are analyzed and what benchmarks are used to determine variance in performance and what actions are taken to address performance issues

5. CAP Consensus Conference: Basic Structure • Survey: • 99 question survey sent to 1245 laboratories participating in PT • Complete data received from 541 laboratories (43% response rate) • Working Groups: • 5 working groups of CT’s and pathologists from variety of practice types

6. CAP Consensus Conference: Working Groups • Interpretive rates, concurrence of CT/CP diagnoses and turnaround time • Prospective and retrospective review • Proficiency testing, general quality and workload • Cytologic-histologic correlation • HPV rates and testing

7. CAP Consensus Conference: Working Groups • Analyzed responses to a portion of the survey • Extensive literature review • Additional questions for clarification posted to CAP web site • Reviewed responses and drafted consensus statements for presentation and voting at consensus conference

8. CAP Consensus Conference: CAP Web Site • Open to entire cytology community • Working groups posted survey results and data thought to generate the most interest • New questions posted and open ended comments solicited • Trends and controversial topics identified • All responses anonymous

9. CAP Consensus Conference • Held on June 4, 2011 in Rosemont, IL • 98 attendees • Working groups presented summary findings and proposed consensus statements for voting • Consensus statements re-worked based on attendee feedback and re-presented • Consensus defined as >50% agreement among attendees

10. CAP Consensus Conference: Post- Conference Work • Data analyzed and formal ranking of degree of consensus performed • Group leaders prepared manuscripts of the body of their work • Manuscripts released together in February 2013 issue of Archives of Pathology and Laboratory Medicine

11. CAP Consensus Conference: Strengths and Limitations • Strengths: number and diversity of participating labs; comprehensive review of gyn cytology QA practices; extensive literature review • Weakness: concerns/sensitivity about potential impact of consensus statements may lead to “watering down” of statements and limited adoption

13. CAP Consensus Conference Working Group 2: Prospective and Retrospective Review • Jennifer Brainard MD, FCAP, Chair • Michael Henry MD, FCAP, Senior Author • George Birdsong MD, FCAP • Tarik Elsheikh MD, FCAP • Kalyani Naik MS, SCT(ASCP) • Margaret Neal MD, FCAP • David Andrew Hartley CT(ASCP)CM, CAP Cytotechnologist Specialist

14. Prospective and Retrospective Review: Definitions • Prospective review • Review, prior to sign-out, by a second cytotechnologist of a subset of Pap tests interpreted as NILM in the first cytotechnologist review • ≠ Prescreen • Retrospective review • Review of NILM (or NILM +) Pap tests that have been signed out: an example is NILM slides from the preceding 5 years in patients with current HSIL+

15. Background: Prospective Review • CLIA 88 – Sec 493.1274 (c)1 • Mandates at least 10% of cases screened by a cytotechnologist be rescreened by a qualified supervisory cytotechnologist or pathologist prior to sign-out • Must be randomly selected and include cases from women at increased risk for developing cervical cancer • Labs must have a written policy that defines high risk and its method of random selection

16. Background: Retrospective Review • CLIA 88 – Sec 493.1274 (c)3 • Patients with current HSIL+ must have their negative cases, if available, from the prior 5 years reviewed • If significant discrepancies are found, which affect current patient care, an amended report must be issued • Serves as a quality monitor and educational tool for the laboratory

17. Background: Results of Reviews • CLIA 88 – Sec 493.1274 (c)4,5,6 and (d) • Records of all rescreening results must be documented • There must be an annual statistical evaluation of the number of reviews of any NILM cases reclassified as LSIL+ • These reviews are compared as individuals against the laboratories’ overall statistics, and are one of the elements used to determine workload limits

18. Background: CAP Checklist • CYP.07478 : 10% Rescreen – same as CLIA and clarifies that slides screened by MD certified in AP and qualified as a technical director do not need to be rescreened • Otherwise, essentially reiterates CLIA regulations

19. Prospective and Retrospective Review • Attempt to evaluate the quality of Pap test screening and interpretation through directed re-examination of Pap test slides • Controversial, emotionally charged and difficult to perform • Results can dramatically impact labs and individuals

20. Prospective and Retrospective Review • Can we increase the value of these metrics for laboratories with a goal of having a positive impact on patient care?

21. Background: Working Group 2 Goal: Discuss the findings of the CAP/CDC project pertinent to prospective and retrospective rescreening and to provide information regarding best laboratory practices based on survey results, consensus conference discussions, published data and expert opinion

22. Prospective Review • Instituted by a number of labs before studies regarding efficacy carried out • Incorporated into the CLIA regulations • Calculated false negative rates (most <5%) are unrealistically low and inaccurate • Error identification expected to be low: non-enriched population • Definition of “high risk” and % high risk cases to include left up to labs • Biased review

23. Prospective Review • False negative rate of the rescreen process far exceeds the false negative rate of primary screening (21% in one study) • Identification of poor screening performance may not occur quickly enough to be of benefit • >90% of errors remain undetected • General consensus: Poor measure of quality

24. Prospective Review: Survey Results ** >85% of labs rescreen >10% NILM cases

25. Prospective Review: Alternatives/ Enhancements • Aimed at increasing thoroughness of rescreening and limiting bias • Rapid prescreening/ rescreening • Enriching the random rescreen population with known SIL • 100% rescreening of NILM Pap tests • Sharing cases among laboratories • Automated quality control slide selection • More realistic measure with more error detection, but quite difficult to implement

26. Prospective Review: Practical Enhancement • Maximize the number of high risk cases by including all readily identifiable high risk cases • Multiple studies: patients with a false negative Pap test who develop HSIL+ usually have a history of an abnormal Pap test and most are hrHPV positive • Pap negative, hrHPV negative patients are at very low risk for progression to HSIL+ • Most labs do not include enough high risk cases in prospective review

27. High Risk Cases in Prospective Review: Survey Results • % High Risk in NILM Cases Rescreened % Cases% Labs 0 2.6% 1-10 51.7% 11-20 18.0% >20 27.8% • Majority of labs (72%) include <20% high risk cases in their prospective rescreen

28. Prospective Review: Maximizing High Risk Cases • Use multiple measures to identify patients at high risk • Remove patients who no longer meet high risk criteria • Suggested criteria: • Prior HSIL cytology • Prior CIN 2+ biopsy result • Recent/ concurrent hrHPV positivity • No screening in past 5 years • Include unsatisfactory Paps

29. Prospective Review: Maximizing High Risk Cases • Consensus achieved: • Labs should make an effort to maximize high risk cases in prospective review and that multiple measures should be used to identify high risk patients (89%) • All readily identifiable high risk cases should be included (80%) • All readily identifiable hrHPV+ NILM Paps should be rescreened (84.5%)

30. Prospective Review: Use of HPV Test Results • 5% to 7% of women 30 years of age and older with NILM Pap tests are hrHPV + and at increased risk of developing cervical cancer • Studies have shown that false negative Paps are associated with a significant rate of hrHPV positivity (80% in one study) • Major barrier to implementation is limited LIS functionality that does not allow timely identification of hrHPV + patients

31. Prospective Review: Unsatisfactory Pap Tests • Patients with unsatisfactory Pap tests are at higher risk for SIL and carcinoma • Higher incidence of abnormalities detected in subsequent Pap following unsatisfactory Pap • Unsatisfactory Paps should be included in high risk population (87% consensus)

32. Removing Patients from High Risk Category: Survey Results • How are patients removed from the high risk category? (n=468) • Never removed 43.4% • Negative Pap test diagnoses 36.5% • Specified time interval 29.7% • 5 years 52.1% • 3 years 20.8% • Majority of respondents (71%) favor removal of patients from “high risk” category (n=45)

33. Removing Patients from High Risk Category • Proposals for removing patients: • Consecutive negative Paps over a specified time interval • Consecutive negative Paps and negative hrHPV test results over a specified time interval • 3 years from last criterion for high risk status • No consensus on criteria for removal

34. Prospective Review: Tracking Results • Both the number of cases rescreened and the number of lesions identified should be monitored for the laboratory and for individuals • CLIA requires tracking LSIL + • Some variation among labs in reporting ASCUS/ASC-H upgrades from NILM

35. Tracking Upgrades: Survey Results • Does the laboratory track the total number of prospective rescreen cases for each individual cytotechnologist? (Required by CLIA and CAP LAP) (n=512) Yes 84.8% No 15.2% If No, why not? (Results from online questions) “Why would I need that information?” “No easy way to do this” “Small lab. Single cytotechnologist” “Smears read off site”

36. Tracking Upgrades: Survey Results • Does the laboratory track the number of lesions identified by prospective rescreen? (Required by CLIA and CAP LAP) (n=513) Yes 78.6% No 21.4% If No, why not? (Results from online questions) “Never occurred to us to do this. Few lesions are ever identified” “What do you mean by “lesions”? We track the number of false negatives”

37. Tracking Upgrades: Survey Results • Upgrade rates from NILM to ECA that are actively monitored (n=425) • ASCUS/ASCH • Labs: 53.4 – 54.8% • CT’s: 73.9 – 77.9% • LSIL, HSIL, AGC, SCC, ADC • Labs: 57.6 – 69.2% • CT’s: 81.2 – 95.3%

38. Prospective Review: Tracking Results • No consensus on monitoring upgrades to ASCUS • Strong majority agreement (98%) that results of prospective review be shared with individuals, including cytotechnologists and cytopathologists at regular intervals

40. Retrospective Review • Less controversial than prospective review • Most laboratories find this educationally valuable • Focus on patients at very high risk for false negative Pap • Goal to improve the screening process by learning from our mistakes

41. Retrospective Review • May reveal systematic problems in the laboratory or with individuals • Allows for implementation of corrective measures including education to improve performance • Criticism: small number of errors detected with this measure; use in monitoring performance of individuals

42. Retrospective Review • Errors detected at significantly higher rate than in prospective review • While the total number of upgraded cases identified is low, the rate of false negative results is substantial, ranging from 10% to 94% in the literature • Review of the literature suggests most false negatives occur within 3 years of current abnormal (86% within 3 years)

43. Retrospective Review • Sherman and Kelly: 53% false negatives (ASCUS threshhold); 25% rate of unsat Paps called NILM • Tabbara and Sidawy: 38% false negatives (ASCUS/AGUS threshhold) • Hatem and Wilbur: 94% false negatives • Jones et al: CAP Q-probe 20.4% false negatives

44. Retrospective Review: Common Findings • Mix of screening and interpretive errors • High rates of poorly prepared smears with compromising factors including obscuring inflammation and blood • A significant number of unsatisfactory Pap tests interpreted as NILM • Presence of few abnormal cells and “under-interpretation” of atypical immature squamous metaplasia

45. Retrospective Review: Survey Results • How many years back are previous negative Paps chosen for retrospective rescreen based on a current HSIL+ Pap? 1-4 years 0.6% 5 years 96.2% All available years 2.5%

46. Retrospective Review: Survey Results • Which diagnostic categories other than HSIL+ or AIS+ prompt a retrospective review of NILM Paps? None 71.4% AGC 15.0% LSIL 8.6% ASC-US 6.8% ASC-H 2.3%

47. Retrospective Review: Enhancement • Use cervical biopsy results of CIN 2+ to trigger retrospective review • This practice would increase the number of patients with significant lesions included in retrospective review • Strong consensus agreement achieved (87%) among conference attendees

48. Retrospective Review: Enhancement • Inclusion of unsatisfactory Paps adds value to retrospective review • Many studies stress the relationship of unsatisfactory Paps to false negatives • Many of the same factors that render a Pap test unsatisfactory are also strongly associated with false negatives • Consensus among conference attendees achieved (76%)

49. Retrospective Review: Tracking Results • Constant monitoring of trends in false negative rates is required • Tracking of upgrades from NILM to LSIL+ required by CLIA • Monitoring of upgrade rates for pathologists very low • Most studies report interpretive errors in which pathologists play a role

50. Tracking Upgrades: Survey Results • Monitoring of upgrade rates (375 labs) • For NILM to HSIL+ Laboratory 72.0% Cytotechnologist 82.7% Pathologist 37.3% • For ASC-US to HSIL+ Laboratory 27.5% Cytotechnologist 35.7% Pathologist 18.1%