Stanford University, Department of Psychiatry and Behavioral Sciences

1. Receiver Operating Characteristic Curve (ROC) Analysis for Prediction StudiesRuth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg Stanford University, Department of Psychiatry and Behavioral Sciences Stanford University School of Medicine Sierra Pacific MIRECC Veterans Affairs Palo Alto Health Care System

2. The Clinical Need forSignal Detection Procedures • Clinical practice is often “hit or miss” therapy • Try one thing, if that does not work, try another • This is frustrating for the patient and expensive • The Goal: find the best treatment for the patient with specific characteristics • New news in psychiatry; old hat in internal medicine

3. Receiver Operating Characteristic Curve (ROC) Analysis • Signal Detection Technique • Traditionally used to evaluate diagnostic tests • Now employed to identify subgroups of a population at differential risk for a specific outcome (clinical decline, treatment response) • Identifies moderators

4. Receiver Operating Characteristic Curve (ROC) Analysis Historical Development

5. ROC Analysis:Historical Development (1) • Derived from early radar in WW2 Battle of Britain to address: Accurately identifying the signals on the radar scan to predict the outcome of interest – Enemy planes – when there were many extraneous signals (e.g. Geese)?

6. ROC Analysis: Historical Development (2) • True Positives = Radar Operator interpreted signal as Enemy Planes and there were Enemy planes (Good Result: No wasted Resources) • True Negatives = Radar Operator said no planes and there were none (Good Result: No wasted resources) • False Positives = Radar Operator said planes, but there were none (Geese: wasted resources) • False Negatives = Radar Operator said no plane, but there were planes (Bombs dropped: very bad outcome)

7. ROC Analysis:Historical Development • Sensitivity = Probability of correctly interpreting the radar signal as Enemy planes among those times when Enemy planes were actually coming • SE = True Positives / True Positives + False Negatives • Specificity = Probability of correctly interpreting the radar signal as no Enemy planes among those times when no Enemy planes were actually coming • SP = True Negatives / True Negatives + False Positives

8. ROC: Prediction of Enemy Planes by RAF Radar Operators

9. Receiver Operating Characteristic Curve (ROC) Analysis Applications: Evaluating Medical Tests

10. ROC Analysis: Evaluating Medical Tests • The evaluation of the ability of a diagnostic test to identify a disease involves considering: • P=Prevalence = occurrence in the population of the outcome of interest (e.g. disease) • True Positives • True Negatives • False Positives • False Negatives • P=Prevalence=True Positives + False Negatives

11. ROC Analysis: Medical Test Evaluation • True Positives = Test states you have the disease when you do have the disease • True Negatives = Test states you do not have the disease when you do not have the disease • False Positives = Test states you have the disease when you do not have the disease • False Negatives = Test states you do not have the disease when you do

12. ROC Analysis: Evaluating Medical Tests • Sensitivity =The probability of having a positive test result among those with a positive diagnosis for the disease • SE = True Positives / True Positives + False Negatives • Specificity = The probability of having a negative test result among those with a negative diagnosis for the disease • SP = True Negatives / True Negatives + False Positives

13. The Basic Tool: 2X2 Sensitivity (SE)=a/P Specificity (SP)=d/P’

14. ROC: GDS (Test) for Diagnosis of Clinically Confirmed Depression

15. Which Test Do You Use: Medical Tests Evaluation • GDS: SE = .80; SP = .85 • Beck Depression Inventory: SE = .85; SP = .75 • Major Depression Inventory = SE = .66; SP = .63

16. ROC Analysis • ROC first calculates Sensitivity and Specificity • Quality Indices measures the quality of the sensitivity and specificity • ROC computes the quality indices for each predictor to find the ones with optimal sensitivity and specificity

17. To Detect the Optimal Sensitivity and Specificity • Depends on the relative CLINICAL importance of false negatives versus false positives. • W=1 means only false negatives matter. • W=0 means only false positives matter. • W=1/2 means both matter equally. • Analytically: Use weighted kappa.

18. ROC Analysis • P = TP + FN P’= 1- (TP + FN) • Q = TP + FP Q’= 1- (TP + FP) • EFF = TP + TN • κ(0.5, 0) = [ (TP + TN) - (TP + FN)(TP+FP) - (1-(TP + FN)(1-(TP + FP))] [1 – (TP + FN)(TP+FP) - (1-(TP + FN))(1-(TP + FP))]

19. ROC Plane and “Curve” ROC “curve” Ideal Point Random ROC (Q,Q) (P,P)

20. Receiver Operating Characteristic Curve (ROC) Analysis Applications Identifying Predictors of Clinical Outcome

21. ROC Analysis: Prediction Studies (Dr. Kraemer) • ROC can identify predictors/characteristics of patients that are at differential risk for a specific outcome of interest. e.g. What are the Characteristics of AD Patients at risk for rapid decline and are high priority for treatment? • What are the clinical predictors of Alzheimer Disease patients who are “good responders” (or “poor responders”) to cholinesterase inhibitor treatments? • Useful in “real world” clinical medicine where multiple variables affect the clinical outcome and patients seldom have one pure diagnosis

22. ROC: Identifying Predictors of an Outcome • 1. ROC relates a predictor (test) to the clinical outcome of interest (Diagnosis/Gold Standard) • 2. ROC searches all predictors and their associated cut-points • 3. ROC determines which predictor and associated cut-point yields the optimal sensitivity and specificity for identifying the outcome of interest yielding two groups at differential risk for the outcome

23. ROC: Identifying Predictors of an Outcome • 4. ROC is an iterative process that is then rerun automatically for each group yielded in Step 3. in order to examine which predictor and associated cut-point may further divide the groups • 5. ROC will keep searching within each group yielded until one of three stopping rules apply (see Stopping rule slide) • 6. ROC thus identifies subgroups of individuals that are at increased risk for the outcome of interest

24. ROC Analysis:Advantages and Disadvantages • No assumptions of normal distribution • Multiple predictors can be evaluated simultaneously • Indicates interactions among predictors • Indicates cut-points on these predictors • Yields clinically relevant information • Non-hypothesis testing • Requires large samples • Capitalizes on chance: needs stringent stopping rule

25. ROC Analysis: Procedure • Start with large sample size • Define the outcome of interest (always binary) • Choose Success/Failure criteria • Select predictor variables of interest (as many as you like) • Run ROC Program that systematically finds best predictors for Success/Failure

26. The Basic Tool: 2X2 Sensitivity (SE)=a/P Specificity (SP)=d/P’

27. ROC: Identifying Predictors & Their Cut-points • Dichotomous Variables such as Gender: • ROC calculates the Se and Sp for Female vs. Male • For Continuous Variables such as Age: • ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

28. ROC: Gender as Predictor ofClinically Confirmed Depression

29. ROC: Identifying Predictors & Their Cut-points • Dichotomous Variables: ROC calculates the Se and Sp for Female vs. Male, Aphasia vs. No Aphasia, etc. • For Continuous Variables such as Age: • ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

30. ROC: Age as Predictor of Clinically Confirmed Depression

31. ROC: Age as Predictor of Clinically Confirmed Depression

32. Receiver Operating Characteristic Curve (ROC) Analysis Conducting the ROC: An Example

33. ROC Analysis: Procedure • Start with large sample size • Define the outcome of interest • Choose Success/Failure criteria • Identify predictor variables of interest • Run ROC Program that systematically finds best predictors for Success/Failure

34. ROC Analysis: Example • Population under investigation: 1, 472 AD patients from 10 Centerswith a 12 month follow-up • Clinically significant outcome:More rapid decline as defined by a loss of 3 or more MMSE points per year, post-visit O'Hara R et al. (2002). Which Alzheimer patients are at risk for rapid cognitive decline? J Geriatr Psychiatry Neurol;15(4):233-8.

35. Predictor Variables • Age-at -patient-visit • Reported age of symptom onset • Gender • Years of education • Ethnicity • MMSE score • Living Arrangement • Presence of Aphasia • Presence of Hallucinations • Presence of Extrapyramidal Signs

37. Stopping Rules • No more possibilities (rare!) • Inadequate sample size • Optimal test (if ‘a priori’) would not have been statistically significant (p<.001)

38. N=61 (12%)P=.02 N=43 (8%)P=.19 N=57 (11%)P=.30 Figure 10.3 N=512 (100%)P=.53 Non-minority Minority N = 191 (37%)P=.25 N = 321 (63%)P=.70 Bayley Mental Dev. Index < 115 Bayley Mental Dev. Index ≥ 115 Mother neverattended college Mother attended college N=110 (21%)P=.48 N=87 (17%)P=.45 N=104 (20%)P=.09 N=211 (41%)P=.81 Bayley Mental Dev. Index<106 Bayley Mental Dev. Index≥106 Bayley Mental Dev. Index<106 Bayley Mental Dev. Index≥106 Attended, didnot graduate Graduatedfrom college N=131 (26%)P=.91 N=80 (16%)P=.65 N=30 (6%)P=.73 ROC Decision Tree for IHDP Control group with outcome of low IQ at age 3. (w= 0.5)

39. ROC Plane and “Swarm” of Points ROC “curve”

40. To Detect the Optimal Sensitivity and Specificity • Depends on the relative CLINICAL importance of false negatives versus false positives. • W=1 means only false negatives matter. • W=0 means only false positives matter. • W=1/2 means both matter equally. • Analytically: Use weighted kappa. • Geometrically: Draw a line through the Ideal Point with slope determined by P and w. Push this line down until it just touches the ROC “curve”. That point is optimal.

41. ROC Analysis: Conclusion • Yields Clinically Relevant Information • Identifies complex interactions • Identifies individuals with different characteristics but at the same risk for the clinically relevant outcome • Identifies individuals at the least risk • Can take differential clinical costs of false positives and false negatives into account

42. Conclusion • It is not sufficient to identify risk factors or even to identify moderators and mediators etc. or a structural model. • It is necessary to present and interpret the results so that clinicians, policy makers, consumers, other researchers can apply them. • ROC trees are one method to accomplish this purpose.

43. Receiver Operating Characteristic Curve (ROC) Analysis Using the ROC Program

44. Using the ROC ProgramA. How to Get the ROC Program • Go to http://mirecc.stanford.edu • Go to “Top Information Requests” • Go to “ROC4 is available for download HERE.” • Double Click on “HERE” • A pop-up window will give you the option to open or save the ROC4 zip file • Best option is to save it to a folder you have already created e.g. ROC analysis

45. Using the ROC ProgramB. Opening the ROC Program • Go to your “ROC analysis” folder • Unzip the ROC4.zipfile (Some computers will automatically unzip when you double click on it or you may need to use an unzip program) • Once unzipped the following 5 files will appear • Read_Me.doc A help file which explains what to do • ROC4.19.exe The actual ROC program • rDemoData.bat Batch file that gets ROC program to run • Demo.txt A demo data input file • runDemoData.doc A demo data output file

46. What the Files Look Like

47. Using the ROC ProgramC. Preparing Data for ROC Program • First prepare your data file • Put your data in Excel form • Your outcome measure should always be: • Dichotomous • Coded as a 1 or 0 • In the far right column • All dichotomous predictor variables coded as a 1 or 0 • All missing data coded as –9999.99 • Remove all IDs or other non-predictor information • Save your Excel data file as Text (Tab delimited) • Give it a name that has no spaces: This will be your data input file

48. What Your Data Input File Looks Like

49. Using the ROC ProgramD. Executing the ROC Program • Open up Microsoft Word • Within Word open the rDemoData batch file • It will open to read as follows echo "Program running- check folder with output file and REFRESH to confirm running “ roc4.19 Demo.txt 50> runDemoData.doc • Where you see “Demo”, you replace with the name of your data input file • Where you see “runDemoData”, you replace with the name of your data output file • Then save your new batch file with a new name and put .bat at the end of the name (easiest name is one associated with the data names you have assigned).

50. Using the ROC ProgramD. An Example of Executing the ROC Program • Helena has data entitled “Workshop” saved as text and now called “Workshopdata.txt” • Within Word open the rDemoData batch file to read as follows echo "Program running- check folder with output file and REFRESH to confirm running “ roc4.19 Demo.txt 50> runDemoData.doc • “Demo” is replaced with “Workshopdata” • “runDemoData” is replaced with “runWorkshopdata” • New batch file is saved as “rWorkshopdata.bat” echo "Program running- check folder with output file and REFRESH to confirm running “ roc4.19 Workshopdata.txt 50> runWorkshopdata.doc • Double Left Click on new batch file and as if by magic your output file entitled “runWorkshopdata.doc” will appear