Observational Study Designs and Studies of Medical Tests

1. ObservationalStudy Designs andStudies of Medical Tests Michael A. Kohn, MD, MPP 25 August 2009

2. Outline • Single Sentence Study Description • REVIEW Observational study designs • Cohort, Double Cohort • Case-Control • Cross-sectional • Studies of Medical Tests • Diagnostic Test Accuracy • Prognostic Test Accuracy • Examples of observational designs (“Name that Design”)

3. Single-Sentence Study Description(Unless Studying a Medical Test) “The [cute acronym] study is a [DESIGN] study of the association* between [predictor] and [outcome] in [study population].” “The SCOTCH Study is a cohort study of the association* between HPV infection and development of cutaneous squamous cell carcinoma in renal transplant recipients.” * Interested in causal association.

4. Single-Sentence DescriptionIf Studying a Test “The [cute acronym] study is a [DESIGN] study of [test] as a [diagnostic/prognostic] test for [disease/outcome] in [study population].” “The 3D-ERUS Study is a cross-sectional study of the accuracy of endorectal ultrasound in re-staging rectal cancer relative to the gold standard of surgical pathology after neoadjuvant chemoradiation in patients with locally invasive rectal cancer.”

5. Single-Sentence Study Description Exercise for section today: Present your study with a sentence like this.

6. Study Design • Not just a matter of semantics • Weaknesses and strengths associated with each study design • Different measures of disease association • Worth getting right or at least thinking about

7. Study Designs • Experimental -- Randomized controlled trial • Observational (today’s topic) -- Cohort -- Double Cohort (exposed-unexposed) -- Case-control -- Cross-sectional

8. Predictor Type and Experimental vs. Observational Design • Predictor = treatment or screening program • -- experiment (randomized controlled trial)* • -- observational study of a treatment or program • Predictor = exposure or risk factor • -- observational study of an exposure or risk factor • Predictor = test result • -- observational study of a test *Not all treatments or screening programs require RCTs to prove effectiveness.

9. OBSERVATIONAL STUDIES • Only option if predictor is a potentially harmful exposure, risk factor, or test. • Even if the predictor is an intervention, RCT may not be feasible • Confounding is an issue* • More intellectually interesting than RCTs? * Except in studies of tests, then the issue isn’t confounding, but how much the test adds to information that is already available.

10. Note on Figures Following schematics of observational study designs assume: • Predictor = Risk Factor • Outcome = Disease

11. Cohort Study

12. Cohort Studies 1)Determine predictor status on a sample from a single population (defined by something other than the predictor). 2)Exclude any potential subjects who already have the outcome. 3)Follow sample over time and attempt to determine outcome on all subjects.

13. Cohort Studies • Can identify individuals lost to follow up • Can estimate overall incidence of outcome in the population (e.g., cases/person-year) • Measure of disease association is the relative risk (RR) or relative hazard (RH)

14. Double Cohort Study

15. Double Cohort (Exposed-Unexposed) Studies • Sample study subjects based on predictor status. • Exclude potential subjects in whom outcome has already occurred. 3) Attempt to determine outcome in all subjects over time.

16. Double Cohort (Exposed-Unexposed) Studies • Can identify individuals lost to follow up • Cannot estimate overall incidence of outcome in the population (e.g., cases/person-year) • Measure of disease association is the relative risk (RR) or relative hazard (RH)

17. Cohort Studies: Sampling Frame vs. Time Frame Time Frame: All cohort studies are longitudinal (follow patients over time). Sampling Frame: Double cohort study -- samples on predictor status Cohort study -- starts with a cross-sectional sample

18. Cohort Studies: Prospective vs. Retrospective Prospective – Predictor status collected as part of this study Retrospective – Predictor status collected by someone else in the past (another study, medical records, etc.) Don’t worry too much about retrospective vs. prospective!

19. Case-Control Study

20. Case-Control Study 1) Separately sample subjects with the outcome (cases) and without the outcome (controls) 2) Attempt to determine predictor status on all subjects in both outcome groups

21. Case-Control Study • Cannot identify individuals lost to follow up (no such thing as “lost to follow up”, since by definition outcome status is known) • Cannot calculate prevalence (or incidence) of outcome • Measure of disease association is the Odds Ratio (OR) • Trying to replicate a nested case control study in which the cases and controls come from the same cohort.

22. Nested Case-Control Study

23. Cross-Sectional Study

24. Cross-Sectional Study Attempt to determine predictor and outcome status on all patients in a single population (defined by something other than predictor and outcome).

25. Cross-Sectional Study • Cannot identify individuals lost to follow up (no such thing as “lost to follow up”) • Can calculate prevalence but not incidence • Measure of disease association is the Relative Prevalence (RP). • Time frame is the same as for a case-control study; both discussed in DCR3, Chapter 8

26. Cohort Studies Start with a Cross-Sectional Study Eliminate subjects who already have disease

27. Causal Association Between Predictor and Outcome • Most observational studies: Does predictor cause outcome? • Studies of diagnostic/prognostic test accuracy: Test result does not cause outcome.

28. Studies of Medical Tests Causality irrelevant. Not enough to show that test result is associated with disease status or outcome*. Need to estimate parameters (e.g., sensitivity and specificity) describing test performance. *Although if it isn’t, you can stop.

29. Studies of Diagnostic Test Accuracy for Prevalent Disease Predictor = Test Result Outcome = Disease status as determined by Gold Standard Designs: Case-control (sample separately from disease positive and disease negative groups) Cross-sectional (sample from the whole population of interest)

30. Dichotomous Tests Sensitivity = a/(a + c) Specificity = d/(b + d)

31. Sensitivity and Specificity Sensitivity PID = “Positive In Disease” Proportion of D+ patients with “+” test result Specificity NIH = “Negative in Health” Proportion of D- patients with “–” test result

32. Studies of Dx Tests Importance of Sampling Scheme If sampling separately from Disease+ and Disease– groups (case-control sampling), cannot calculate prevalence, positive predictive value, or negative predictive value.

33. Dx Test:Case-Control Sampling Sensitivity = a/(a + c) Specificity = d/(b + d)

34. Dx Test: Cross-sectional Sampling Prevalence = (a + c)/N Positive Predictive Value = a/(a + b) Negative Predictive Value = d/(c + d)

35. Immunohistochemical Test for ARVC* *N Engl J Med. 2009 Mar 12;360(11):1075-84.

36. Immunohistochemical Test for ARVC* Your patient has a negative result on this test. Does the NPV of 90% mean he still has a 10% chance of ARVC? *N Engl J Med. 2009 Mar 12;360(11):1075-84.

37. Sample Size Calculations for Studies of Diagnostic Test Accuracy • Sensitivity and Specificity are descriptive proportions.* • Choose N with disease to estimate sensitivity with the desired precision. • Choose N without disease to estimate specificity with the desired precision. *Table 6E, page 91 DCR3

38. Likelihood Ratio LR(result) = P(result|D+)/P(result|D-) P(Result) in patient WITH disease ---------------------------------------------------- P(Result) in patients WITHOUT disease See DCR3, Chapter 12, page 191

39. Sample Size Calculations for Studies of Diagnostic Test Accuracy Size the sample to estimate a likelihood ratio with the desired precision. See DCR3, Chapter 12, page 191

40. Studies of Prognostic Tests for Incident Outcomes Predictor = Test Result Development of outcome or time to development of outcome. Design: Cohort study

41. Studies of Prognostic Tests for Incident Outcomes Prognostic test “result” is often a probability of having the outcome by a future time point (e.g. risk of death or recurrence by 5 years). Need to assess both calibration and discrimination.

42. Comparing Predictions • Evidence-Based Diagnosis, Chapter 7 • Jan. 30, 2008 Issue of Statistics in Medicine* *Pencina et al. Stat Med. 2008 Jan 30;27(2):157-72;

43. Examples Name that observational study design

44. JIFee Babies born at Kaiser with neonatal hyperbilirubinemia (Bili > 25) are compared with randomly selected “controls” from the same birth cohort. Outcome measure is IQ and neurologic status at age 5 years. No difference in IQ or fraction with neurologic disability between the “case” and “control” groups. Newman, T. B., P. Liljestrand, et al. (2006). N Engl J Med354(18): 1889-900.

45. JIFee Design? (Be Careful)

46. RRISK(Reproductive Risk Factors for Incontinence at Kaiser) • Random sample of 2100 women aged 40-69 yo • Interview, self report, diaries to determine whether they have the outcome, urinary incontinence. • Chart abstraction of L&D/surgical records to establish predictor status

47. RRISK Design?

48. HIV Tropism and Rapid Progression* Is HIV CXCR4 (as opposed to CCR5) tropism a predictor of rapid progression in acutely infected HIV patients? Molecular tropism assayis “high end” and labor-intensive. Have funding to perform a total of 80 assays. UCSF OPTIONS cohort follows patients acutely infected with HIV. Has banked serum from near time of acute infection. * Vivek Jain’s Project

49. HIV Tropism and Rapid Progression (continued) Identify the 40 patients with the most rapid progression (Group 1) and randomly select 40 others from the UCSF Options cohort (Group 2). Run the tropism assay on banked serum for these 80 patients and compare results between Group 1 and Group 2.

50. HIV Tropism and Rapid Progression Design?