Julian Tilbury Peter Van Eetvelt John Curnow Emmanuel Ifeachor

1. Objective Evaluation of Intelligent Medical Systems using a Bayesian Approach to Analysis of ROC Curves Julian Tilbury Peter Van Eetvelt John Curnow Emmanuel Ifeachor

2. Contents • Evaluation Problem • Introduction to ROC Curves • Frequentist Approach • Bayesian Approach • Area under the Curve (AUC) • Parametric ROC Curves • Conclusion

3. Evaluation Problem • Collecting Medical Test Cases is Expensive • Desirable to test Systems with few cases • System may Pass by Luck • Must use ‘Confidence Intervals’ • ROC curves - convenient existing representation for results

4. Introduction to ROC Curves • Two populations • Healthy • Diseased • Known by a Gold Standard • Differentiate using a single Test Measure • What Threshold will separate them?

11. Frequentist Approach • E.g. Green & Swets – for each point • False Alarm Rate Confidence Interval • Hit Rate Confidence Interval Combined to give cross

13. Three ‘Problems’ • False Alarm Rate Confidence Interval of Point 0 is zero width • Hit Rate Confidence Interval of Point 1 is zero width • Hit Rate Confidence Interval is beyond the graph • Given the data, this makes no sense!

14. Four Observations • Sample too small • Hit Rate (or False Alarm Rate) near 0 or 1 • Correct within paradigm • Population mean = Sample mean • Distribution of re-sampling • Confidence Interval off Graph • Off-graph = no samples, so add to taste

15. Bayesian Approach • Consider just the False Alarm Rate • Using Bayes’ Law • Assume a prior distribution for the population • Update the distribution according to evidence to give posterior distribution • Combine False Alarm Rate and Hit Rate to give combined posterior distribution • Compute using Dirichlet Integrals

17. (For Point 0)

19. Convergence At low sample sizes the two paradigms give radically different results As the sample size increases the resultant distributions merge Take multiples of 3 False positive and 2 True negatives …

24. Area Under the Curve • Single value used as a summary of diagnostic accuracy • Novel Bayesian method (by Dynamic Programming) • Existing Frequentist methods

27. Parametric ROC Curves • Both Healthy and Diseased populations are ‘Gaussian’ • Curve can be characterised by two parameters: • Difference in Means • Ratio of Standard Deviations

28. 2δh Healthy Sd = δh+ δd 2δd Disease Sd = δh+ δd ( ) 2µh - 2µd Sigmoid Healthy Mean – Disease Mean = δh+ δd

32. Parametric Analysis • Existing Maximum Likelihood • Brittle • Frequentist Confidence Intervals • Novel Analysis (by Dynamic Programming) • Robust • Maximum Likelihood • Posterior Interval for Parameters • and Area Under Curve

36. Parametric Nonparametric

37. Conclusion • Frequentist (for low sample size) • Best – counterintuitive • Worst – ‘wrong’ • Bayesian • Best – robust and accurate • Worst – slow to calculate • Still need the prior distribution • Converge at high sample size • Therefore use Bayesian for all sample sizes