Testing the Limits of a QSAR Model:

1. Testing the Limits of a QSAR Model: How many cases are actually needed to develop a reliable predictive model? C. Matthew Sundling, Curt M. Breneman, Mark J. Embrechts, Changjian Huang, Xiaohua Wu, N. Sukumar April 9th, 2008

2. Related RECCR Presentations • Dr. Dominic Ryan - Stability of rank order predication models of top ten testing molecules QSAR model stability: How much information is in the data? (COMP, Morial Convention Center rm. 347, Wednesday, 3:05 pm) • Prof. Mark Embrechts - One-class SVM for outlier detection and applicability of a model to a specific testing set. Testing the validity range of QSAR models using one-class support vector machines. (COMP, Morial Convention Center rm. 347, Thursday, 1:30pm)

3. Objective • How do models function as training data is reduced? • Hypothesis: if (1) a model is "stable", and the (2) descriptors are appropriate to the effect being modeled, then a great deal of training information could be removed without the testing predictions to degrade significantly.

4. Datasets BP - boiling points ACE - Angiotensin-Converting Enzyme inhibitors AChE - acetylcholinesterase inhibitors Lombardo - blood-brain barrier (BBB) partitioning Artemisinin - anti-malarial compounds (298 compounds) (112 compounds) (60 compounds) (70 compounds) (179 compounds) difficulty

5. Descriptors MOE - “classic” 2D descriptors TAE - electron density derived surface property distributions SS - surface statistics of TAE property distributions Wavelets - alternative representation of TAE information PEST - shape-property 3D hybrid of TAE property distributions ALL - combination of all descriptors

6. Testing Procedure Training Set (70%) Training Set Dataset 90% Training Set Subset Testing Set (30%) Testing Set (30%) Testing Set (30%) PLS models of five components were used throughout the study.

7. Typical Results AChE BP Artemisinin ACE

8. Repeating Testing Procedure Training Set Training Set Training Set Training Set (70%) Training Set Training Set Dataset 90% Training Set Training Set Training Set Subset Training Set Testing Set (30%) Testing Set (30%) Testing Set (30%)

9. Multiple Training Sets

10. Performance Instability

11. Multiple Testing Sets Dataset Training Set (70%) Training Set (70%) Training Set (70%) … Testing Set (30%) Testing Set (30%) Testing Set (30%) … Repeat modeling study …

12. Multiple Testing Sets

13. Distance(trainingset,testset) Can I understand more about the relationship between the training data and the testing data?

14. Distance(trainingset,testset) Distance function = Sum of Euclidean distances for testing molecules to nearest neighbor training molecule

15. Distance(trainingset,testset)

16. Distance(trainingset,testset)

17. Conclusion: Stable?

18. Conclusion: Data vs. Information?

19. Conclusion: Potential applications? Extend ensemble models to include stability analysis?

20. Conclusion: Your model’s performance is dependent on your training data. (Duh!) It’s hard to know when you have enough.

21. Thanks! Curt M. Breneman Mark J. Embrechts Changjian Huang Xiaohua Wu N. Sukumar