Critical appraisal of a diagnostic paper

1. Critical appraisal of a diagnostic paper ShakilaThangaratinam Professor of Maternal and Perinatal Health Women’s Health Research Unit R & D Director of Women’s Health Barts Health NHS Trust

2. Relation of diagnosis and therapy in clinical practice

3. A hierarchical model for evaluation of tests • Assessment of the reliability and other technical aspects of a test • Assessment of the diagnostic accuracy • Assessment of the diagnostic effectiveness and cost effectiveness Fryback et al. Med Decis Making 1991;11:88-94

4. How does a diagnostic test help? A test is useful if it changes our ability to predict whether the patient has the disease or not • The likelihood of disease before the test should be altered by the test result • The likelihood of disease after the test should be different compared to the prevalence of disease in the population

5. Value of a test Likelihood of TEST Likelihood of disease before test RESULT disease after test Prevalence of disease + Should raise the (e.g. 7-11% of pregnant likelihood to ~100% women have preterm delivery )— Should lower the likelihood to ~ 0%

6. How to do systematic reviews • Formulate clear clinical questions from our knowledge needs identified in patient encounters • Search the literature to identify relevant articles • Critically appraise the evidence for its validity and usefulness • Synthesis the evidence • Implement useful findings in clinical practice

7. Critical Appraisal of the Medical Literature • Are the results of the study valid? • What are the results? • Will the results help in caring for patients? Oxman et al. JAMA 1993;270:2093-5

8. A test accuracy study

9. Are the results valid? • Was there an independent “blind” comparison with a reference standard? • Did the patient sample include an appropriate spectrum of patients to whom the test will be applied in clinical practice? • Did the results of the test influence the decision to perform the reference standard?

10. Are the results valid?

11. Hierarchy of evidence for test accuracy studies 1 An independent, blind comparison with reference standard among an appropriate population of consecutive patients. 2 An independent, blind comparison with reference standard among an appropriate population of non-consecutive patients or confined to a narrow population of study patients. 3 An independent, blind comparison among an appropriate population of patients, but reference standard not applied to all study patients. 4 Non-blind comparison or reference standard not applied independently 5 Expert opinion with no explicit critical appraisal, based on physiology, bench research or first principles

12. Critical Appraisal of the Medical Literature • Are the results of the study valid? • What are the results? • Will the results help in caring for patients? Oxman et al. JAMA 1993;270:2093-5

13. What are the results? What is the “benefit” or “added value” of test as a diagnostic aid? Can we measure this added benefit? • Are clinically meaningful measures of diagnostic accuracy provided? • Are the data necessary to calculate the diagnostic accuracy measures provided?

14. Consequences of testing

15. Measures of test accuracy • Sensitivity and Specificity • Predictive values • Likelihood ratios • Diagnostic odds ratio

16. Sensitivity

17. Sensitivity Sensitivity is the proportion of those people who really have the disease (TP+FN) who are correctly identified as such (TP) • Sensitivity = TP/(TP+FN) Disease Present Absent Positive Test Negative

18. Sensitivity • Sensitivity is the proportion of those people who really have • the disease (TP+FN) who are correctly identified as such (TP) • Sensitivity = TP/(TP+FN) Disease Present Absent Positive Test Negative

19. Specificity

20. Specificity • Specificity is the proportion of those people who really do not • have the disease (TN+FP) who are correctly identified as such (TN) • Specificity = TN/(TN+FP) Disease Present Absent Positive Test Negative

21. Specificity • Specificity is the proportion of those people who really do not • have the disease (TN+FP) who are correctly identified as such (TN) • Specificity = TN/(TN+FP) Disease Present Absent Positive Test Negative

22. Predictive Values

23. PPV • Positive Predictive Value is the proportion of the people who test • positive (TP+FP) who truly have the disease (TP) • Positive predictive value = TP/(TP+FP) Disease Present Absent Positive Test Negative

24. PPV • Positive Predictive Value is the proportion of the people who test • positive (TP+FP) who truly have the disease (TP) • Positive predictive value = TP/(TP+FP) Disease Present Absent Positive Test Negative

25. NPV Negative Predictive Value is the proportion of the people who test negative (TN+FN) who truly do not have the disease (TN) Negative predictive value = TN/(TN+FN) Disease Present Absent Positive Test Negative

26. NPV Negative Predictive Value is the proportion of the people who test negative (TN+FN) who truly do not have the disease (TN) Negative predictive value = TN/(TN+FN) Disease Present Absent Positive Test Negative

27. Problems • Sensitivity and specificity are characteristics of the test • Predictive values are dependent on the prevalence of the disease • Our population is often quite different from the study population

28. Search bibliographic databases Design search strategy Formulate research question The systematic review process Further selection of primary studies using inclusion criteria Identify possible papers from titles/abstracts Retrieve papers Formulate research / policy conclusions Quality appraisal Synthesis Extract data