1. Reference standard Diagnosis: the pathway of a diagnostic test From bench to bedside Gennaro D’Amico UOC Gastroenterologia Ospedale V Cervello– Palermo gedamico@libero.it

2. Terminology In diagnosis research, the Reference Standard (RS) is the procedure (or test) that is used to define the true state of the patient (disease vs no disease) A major aim of diagnosis research is to find new diagnostic tests (Index Test) (IT) less invasive and expensive than the RS

3. The index test (IT) Positive Disease hypothesis Test under study Negative

4. The question underlying test accuracy research Are these patients truly Positive With disease ? True Positive Test under study Negative Free of disease True Negative To answer, a verification test is needed: the RS

5. The ideal (perfect) reference (gold) standard RS • “An ideal RS, in an optimal diagnostic accuracy study, would fulfill the following criteria: • The RS provides error-free classification of allsubjects. • (2) The same RS is used to verify all ITresults. • (3) The IT and RS can be performed within a short interval to avoid changes in target condition status″ Retisma JB. J Clin Epidemiol 2009; 62: 787-806

6. Verification of a new test accuracy Reference standard Negative Positive No disease Disease Positive true Test under study false Negative true false

7. Imperfect RS • A perfect gold standard with 100% sensitivity and specificity is exceptional • The clinical condition to which the IT is to be applied may hamper the application of the (same) RS in all the subjects • The time interval from IT to RS may be too long • A satisfactory RS may not be available For these reasons the term reference standard is preferred to gold standard

8. Problems related to imperfect RS Retisma JB. J Clin Epidemiol 2009; 62: 787-806 Whiting P. Ann Int Med 2004;140:189-202

9. Different RSdiagnosis of depression • IT: TRH stimulation test for depression (TSH < 7 µIU/ml post-TRH infusion of 500 p.g IV). • Ten sensitivity studies used two different RSs, based on different validated questionnaires • Diagnostic and Statistical Manual III criteria (DSM III), American PsychiatryAssociation (APA, 1980) • Research Diagnostic Critera (RDC) (Spitzer, 1978) Arana GW. Biol Psych 1990;28:733-737

10. Different RSsSensitivity of TRH-ST for depression Applying different standard procedures for different patients may yield inconsistent reference for the IT as each of the «standards» will have its own error rate DSM RDC 70 60 50 Sensitivity % 40 30 20

11. Inappropriate RSsensitivity of excercise scintigraphy for coronary disease Planar vs TC coronarographyas RS Planar TC Detrano R. Arch Int Med 1988;148:1289-1295

12. Courtesy of dr Mirella Fraquelli Trade off Sensitivity/Specificity Imperfect RS vs truedisease status IT Broad histopathologic criteria to diagnose colon dysplasia Liver biopsy in diagnosing hepatic fibrosis Specificity Sensitivity Underestimates specificity Underestimates sensitivity

13. Verification biasRF is not performed in all subjects or different RSs are used Partialverification: • RS performed on test-positives, but not on test-negatives Differentialverification: • RS used for test-positives is different from thatused for test-negatives

14. Prospective investigation of Pulmonary Embolism Investigation PIOPED • Diagnostic accuracy of ventilation perfusion scan was assessed by angiography • Angio was more commonly done in patients with higher probability of PE based on VQ scan results Partial verification bias JAMA 1990;263:2753-59

15. Independent assessment of RS Diagnosticreview bias Test under study Reference standard blind unblind + - RS test review bias FP + - TP Test under study FN TN

16. Incorporation bias • The test that is being evaluated is included in the RS • It can lead to overestimation of test accuracy • It can occur if final diagnosis is made on the basis of all clinical data (which might include the IT) • Examples: PCR for tuberculosis, Mantoux for TB among kids, screening for depression

17. Possible solutions for imperfect or missing RS

18. Solving problems of imperfect RSDelayed-type cross-sectional study • RS is invasive and maynot be performed in allsubjects or differentstandardshave to be used • The disease state of includedsubjectsmay be verifiedafter a predefined follow-up Knotterus JA. J Clin Epidemiol 2003;56:118-1128

19. Prospective investigation of Pulmonary Embolism Investigation PIOPED • Diagnostic accuracy of ventilation perfusion scan was assessed by angiography • Angio was more commonly done in patients with higher probability of PE based on VQ scan results • 1-year follow-up confirmed the diagnosis in subjects not undergoing angio JAMA 1990;263:2753-59

20. Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital Index test in patients with suspected PE Verification Angio TC V/Q pulmonary scinti-scan Angiography Leg venous US showing DVT • Wells score >4 PE likely ≤ 4 PE unlikely Posadas-Martines ML Thromb Res 2014;133:177–181

21. Performance of the Wells score in patients with suspected pulmonary embolism during hospitalization: A delayed-type cross sectional study in a community hospital 613 pts Suspected PE 90 days followup +1 PE No PE Posadas-Martines ML Thromb Res 2014;133:177–181

22. Validation of IT in the absence of satisfactory RSInterferon gamma levels for latent tuberculosis The better test would be the one more strongly correlated with exposure E= IFN-Ỿ T= tubercolin skin test Ewer K Lancet 2003;361:1168-73

23. Conclusions • The RS is the diagnostic instrument used to verify the accuracy of the results of a new test • The same RS should independently, or even blindly,verify all the new test results and should be correctly performed • Since the RS is almost always imperfect it should be appropriate for the new test under study • Imperfect RS or inappropriate use of satisfactory RS may leed to incorrect conclusions on the IT accuracy • Composite RS, expert panel judgement, cross-sectional delayed verification and other validation methods, may help to overcome the lack of a satisfactory RS

24. Diagnosticreview bias Independent assessment of RS Test under study Reference standard unblind blind RS test review bias + - TP FP + - Test under study FN TN