1. Classification Derek Hoiem CS 598, Spring 2009 Jan 27, 2009

2. Outline • Principles of generalization • Survey of classifiers • Project discussion • Discussion of Rosch

3. Pipeline for Prediction Imagery Representation Classifier Predictions

4. Free Lunch Theorem

5. Bias and Variance Error High Bias Low Variance Complexity Low Bias High Variance

6. Overfitting • Need validation set • Validation set not same as test set

7. Bias-Variance View of Features • More compact = lower variance, potentially higher bias • More features = higher variance, lower bias • More independence among features = simpler classifier  lower variance

8. How to reduce variance • Parameterize model E.g., linear vs. piecewise

9. How to measure complexity? • VC dimension Upper bound on generalization error Training error + N: size of training set h: VC dimension : 1-probability

10. How to reduce variance • Parameterize model • Regularize

11. How to reduce variance • Parameterize model • Regularize • Increase number of training examples

12. Effect of Training Size Error Number of Training Examples

13. Risk Minimization • Margins x x x x x x x o x o o o o x2 x1

14. Classifiers • Generative methods • Naïve Bayes • Bayesian Networks • Discriminative methods • Logistic Regression • Linear SVM • Kernelized SVM • Ensemble methods • Randomized Forests • Boosted Decision Trees • Instance based • K-nearest neighbor • Unsupervised • Kmeans

15. Components of classification methods • Objective function • Parameterization • Regularization • Training • Inference

16. Classifiers: Naïve Bayes • Objective • Parameterization • Regularization • Training • Inference y x1 x2 x3

17. Classifiers: Logistic Regression • Objective • Parameterization • Regularization • Training • Inference

18. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference x x x x x x x o x o o o o x2 x1

19. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference x x x x x x x o x o o o o x2 x1

20. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference Needs slack x x o x x x x x o x o o o o x2 x1

21. Classifiers: Kernelized SVM • Objective • Parameterization • Regularization • Training • Inference x x o o o x x x1 x x o o x12 o x x x1

22. Classifiers: Decision Trees • Objective • Parameterization • Regularization • Training • Inference x x x x x x o x o x o o o o x2 x1

23. Ensemble Methods: Boosting figure from Friedman et al. 2000

24. Boosted Decision Trees High in Image? Gray? Yes No Yes No Smooth? Green? High in Image? Many Long Lines? … Yes Yes No Yes No Yes No No Blue? Very High Vanishing Point? Yes No Yes No P(label | good segment, data) Ground Vertical Sky [Collins et al. 2002]

25. Boosted Decision Trees • How to control bias/variance trade-off • Size of trees • Number of trees

26. K-nearest neighbor • Objective • Parameterization • Regularization • Training • Inference x x o x x x x o x o x o o o o x2 x1

27. Clustering + x + o + + x x + x + + x + + o x x + o + o + o x2 x2 x1 x1

28. References • General • Tom Mitchell, Machine Learning, McGraw Hill, 1997 • Christopher Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995 • Adaboost • Friedman, Hastie, and Tibshirani, “Additive logistic regression: a statistical view of boosting”, Annals of Statistics, 2000 • SVMs • http://www.support-vector.net/icml-tutorial.pdf

29. Project ideas?

30. Discussion of Rosch