Learning with Trees

1. Learning with Trees Rob Nowak University of Wisconsin-Madison Collaborators: Rui Castro, Clay Scott, Rebecca Willett www.ece.wisc.edu/~nowak Artwork: Piet Mondrian

2. Basic Problem: Partitioning Many problems in statistical learning theory boil down to finding a good partition function partition

3. Classification Learning and Classification: build a decision rule based on labeled training data Labeled training features Classification rule: partition of feature space

4. Signal and Image Processing MRI data brain aneurysm Extracted vascular network Recover complex geometrical structure from noisy data

5. Partitioning Schemes Support Vector Machine image partitions

6. Why Trees ? • Simplicity of design • Interpretability • Ease of implementation • Good performance in practice Trees are one of the most popular and widely used machine learning / data analysis tools CART:Breiman, Friedman, Olshen, and Stone, 1984 Classification and Regression Trees C4.5: Quinlan 1993, C4.5: Programs for Machine Learning JPEG 2000: Image compression standard, 2000 http://www.jpeg.org/jpeg2000/

7. Example: Gamma-Ray Burst Analysis photon counts burst Compton Gamma-Ray Observatory Burst and Transient Source Experiment (BATSE) time One burst (10’s of seconds) emits as much energy as our entire Milky Way does in one hundred years ! x-ray “after glow”

8. fine partition Trees and Partitions coarse partition

9. Estimation using Pruned Tree piecewise constant fits to data on each piece of the partition provides a good estimate Each leaf corresponds to a sample f(ti ), i=0,…,N-1

10. Gamma-Ray Burst 845 piecewise linear fit on each cell piecewise polynomial fit on each cell

11. Recursive Partitions

12. Adapted Partition

13. Image Denoising

14. Decision (Classification) Trees Bayes decision boundary labeled training data complete partition pruned partition decision tree - majority vote at each leaf

15. Classification Ideal classfier Adapted partition histogram 256 cells in each partition

16. Image Partitions 1024 cells in each partition

18. Image Coding JPEG 0.125 bpp JPEG 2000 0.125 bpp non-adaptive partitioning adaptive partitioning

19. Probabilistic Framework

20. Prediction Problem

21. Challenge

22. Empirical Risk

23. Empirical Risk Minimization

24. Classification and Regression Trees

25. Classification and Regression Trees 1 1 1 1 1 1 0 1 1 0 0 0 0 1 0 0 0 0 0

26. Empirical Risk Minimization on Trees

27. Overfitting Problem crude stable accurate variable

28. Bias/Variance Trade-off large bias small variance coarse partition small bias large variance fine partition

29. Estimation and Approximation Error

30. Estimation Error in Regression

31. Estimation Error in Classification

32. trust no trust risk variance overfitting to data Partition Complexity and Overfitting empirical risk # leaves

33. Controlling Overfitting

34. Complexity Regularization

35. Per-Cell Variance Bounds: Regression

36. Per-Cell Variance Bounds: Classification

37. Variance Bounds

38. A Slightly Weaker Variance Bound

39. “small” leafs contribute very little to penalty Complexity Regularization

40. Example: Image Denoising This is special case of “wavelet denoising” using Haar wavelet basis

41. Theory of Complexity Regularization

42. Coffee Break !

43. DALI ? mustache eyes Classification

44. Probabilistic Framework

45. Learning from Data 0 1 0 1

46. Approximation and Estimation 0 Approximation 1 BIAS Model selection VARIANCE

47. Classifier Approximations 0 1

48. Approximation Error Symmetric difference set Error

49. Approximation Error boundary smoothness risk functional (transition) smoothness

50. Boundary Smoothness