Constantin F. Aliferis M.D., Ph.D., FACMI

1. Introduction to translational and clinical bioinformaticsConnecting complex molecular information to clinically relevant decisions using molecular profiles Alexander Statnikov Ph.D. Director, Computational Causal Discovery laboratory Assistant Professor, NYU Center for Health Informatics and Bioinformatics, General Internal Medicine Constantin F. Aliferis M.D., Ph.D.,FACMI Director, NYU Center for Health Informatics and Bioinformatics Informatics Director, NYU Clinical and Translational Science Institute Director, Molecular Signatures Laboratory, Associate Professor, Department of Pathology, Adjunct Associate Professor in Biostatistics and Biomedical Informatics, Vanderbilt University

2. Key Principles for Developing Robust Molecular Signatures Use supervised method to select genes relevant to prediction of the phenotype Use supervised method to build molecular signature/profile (or classification model) for prediction of the phenotype Steps #1 and #2 should be applied by cross-validation such that testing data is used only once for estimation of predictive accuracy. I.e., testing data is neither used for gene selection nor for building of classification model.

3. Principle #1:Use supervised method for gene selection

4. Select genes to predict patients who will survive > 5 yr Will select these 3 genes!

5. Select genes to predict patient who will respond to treatment Will select these 3 genes!

6. Principle #2:Use supervised method for classification (development of molecular signatures or profiles)

7. Input data for two genes p53 Rb

8. We apply clustering algorithm that finds 2 clusters p53 Rb

9. Unfortunately clustering is a non-specific method and falls into the ‘one-solution fits all’ trap when used for prediction Do not Respond to treatment Tx2 p53 Rb Respond to treatment Tx2

10. Principle #3:Gene selection and training of the classifier should be performed by cross-validation, such that testing data is used only once for estimation of predictive accuracy.

11. Hold-out validation method data test train

12. N-Fold Cross-validation data train test train train train train train test test test test test 0.9 0.8 0.8 0.9 0.8 0.9 Average accuracy = 0.85

13. Repeated N-Fold Cross-validation Average accuracy = 0.85 data Average accuracy = 0.8 Average accuracy (over different splits into N folds) = 0.87 Average accuracy = 0.9

14. Leave-one-out cross-validation Each test set consists of a one sample data train test train train train train train test test test test test Leave-one-out cross-validation (LOOCV) is equivalent to N-fold cross-validation, where N is equal to the number of samples in the dataset.

15. Simon’s Experiment

16. Avoid training and testing on the same subject A(sample #1) B(sample #1) C(sample #1) D(sample #1) A(sample #2) B(sample #2) C(sample #2) D (sample #2) A(sample #3) B (sample #3) C (sample #3) D (sample #3) A (sample #4) B (sample #4) C (sample #4) D (sample #4) A(sample #1) B(sample #1) C(sample #1) D(sample #1) A(sample #2) B(sample #2) C(sample #2) D (sample #2) A(sample #3) B (sample #3) C (sample #3) D (sample #3) A (sample #4) B (sample #4) C (sample #4) D (sample #4) A(sample #1) A(sample #2) A (sample #3) A (sample #4) B(sample #1) B(sample #2) B (sample #3) B (sample #4) C(sample #1) C(sample #2) C (sample #3) C (sample #4) D(sample #1) D(sample #2) D(sample #3) D(sample #4) Train Train Test Test Unbiased CV Biased CV

17. Developing molecular profiles for diagnosis of acute respiratory infections Alexander Statnikov, Nikita Lytkin, Lauren McVoy, Constantin Aliferis

18. Experimental design of Zaas et al. Asymptomatic Symptomatic time Baseline (everybody is healthy) Baseline + ε (injection of virus) Peak (for observing symptoms)

19. Experimental design of Zaas et al. Asymptomatic Asymptomatic at baseline Symptomatic Symptomatic at baseline time Baseline (everybody is healthy) Baseline + ε (injection of virus) Peak (for observing symptoms)

20. Data analysis tasks Asymptomatic at baseline Asymptomatic at peak Task I(a) and (b) Symptomatic at baseline Symptomatic at peak time Baseline (everybody is healthy) Baseline + ε (injection of virus) Peak (for observing symptoms)

21. Data analysis tasks Asymptomatic at baseline Asymptomatic at peak Task IV (a) and (b) Task II Task III Symptomatic at baseline Symptomatic at peak time Baseline (everybody is healthy) Baseline + ε (injection of virus) Peak (for observing symptoms)

22. Data analysis tasks Previous results from Zaas et al: AUC of the signature for task 1a (with inappropriate gene selection using all data) estimated by LOOCV= 0.983 When the signature for task 1a is applied to independent data (but only with one type of virus), its AUC = 1.000 Were unable to find a signature to distinguish groups in tasks 2-4.

23. Our results in the data of Zaas et al.using linear SVM classifiers Zaas et al. could not find signal here!

24. A closer examination of task II with non-linear classifiers I.e., no predictive signal can be found!

25. Validation of our results in the independent data from Ramilo et al. for signature from task I Biased CV: Task I(a) | Unbiased CV: Task I(b)

26. Statistical comparison with the signature of Zaas et al. for task I

27. Visualization of our signature for task I

28. Genes in our signature for task I

29. Genes in our signature for task III

30. Genes in our signature for task IV

31. Why Zaas et al. could not find signatures for tasks III and IV? Most plausible explanation so far is that they used genes obtained from supervised gene selection method for a different classification task!

32. Next time We will be building and validating molecular signatures for acute respiratory viral infections using GEMS software