Support Vector Machines: Kernel Embedding Visualization

1. Object Orie’d Data Analysis, Last Time • Kernel Embedding • Embed data in higher dimensional manifold • Gives greater flexibility to linear methods • Support Vector Machines • Aimed at very non-Gaussian Data • E.g. from Kernel Embedding • Distance Weighted Discrimination • HDLSS Improvement of SVM

2. Support Vector Machines Graphical View, using Toy Example: • Find separating plane • To maximize distances from data to plane • In particular smallest distance • Data points closest are called support vectors • Gap between is called margin

3. Support Vector Machines Graphical View, using Toy Example:

4. Support Vector Machines Forgotten last time, Important Extension: Multi-Class SVMs Hsu & Lin (2002) Lee, Lin, & Wahba (2002) • Defined for “implicit” version • “Direction Based” variation???

5. Support Vector Machines Also forgotten last time, Toy examples illustrating Explicit vs. Implicit Kernel Embedding As well as effect of window width, σ on Gaussian kernel embedding

6. SVMs, Comput’n & Embedding For an “Embedding Map”, e.g. Explicit Embedding: Maximize: Get classification function: • Straightforward application of embedding • But loses inner product advantage

7. SVMs, Comput’n & Embedding Implicit Embedding: Maximize: Get classification function: • Still defined only via inner products • Retains optimization advantage • Thus used very commonly • Comparison to explicit embedding? • Which is “better”???

8. Support Vector Machines Target Toy Data set:

9. Support Vector Machines Explicit Embedding, window σ = 0.1:

10. Support Vector Machines Explicit Embedding, window σ = 1:

11. Support Vector Machines Explicit Embedding, window σ = 10:

12. Support Vector Machines Explicit Embedding, window σ = 100:

13. Support Vector Machines Notes on Explicit Embedding: • Too small  Poor generalizability • Too big  miss important regions • Classical lessons from kernel smoothing • Surprisingly large “reasonable region” • I.e. parameter less critical (sometimes?) Also explore projections (in kernel space)

14. Support Vector Machines Kernel space projection, window σ = 0.1:

15. Support Vector Machines Kernel space projection, window σ = 1:

16. Support Vector Machines Kernel space projection, window σ = 10:

17. Support Vector Machines Kernel space projection, window σ = 100:

18. Support Vector Machines Kernel space projection, window σ = 100:

19. Support Vector Machines Notes on Kernel space projection: • Too small  • Great separation • But recall, poor generalizability • Too big  no longer separable • As above: • Classical lessons from kernel smoothing • Surprisingly large “reasonable region” • I.e. parameter less critical (sometimes?) Also explore projections (in kernel space)

20. Support Vector Machines Implicit Embedding, window σ = 0.1:

21. Support Vector Machines Implicit Embedding, window σ = 0.5:

22. Support Vector Machines Implicit Embedding, window σ = 1:

23. Support Vector Machines Implicit Embedding, window σ = 10:

24. Support Vector Machines Notes on Implicit Embedding: • Similar Large vs. Small lessons • Range of “reasonable results” Seems to be smaller (note different range of windows) • Much different “edge” behavior Interesting topic for future work…

25. Distance Weighted Discrim’n 2-d Visualization: Pushes Plane Away From Data All Points Have Some Influence

26. Distance Weighted Discrim’n References for more on DWD: • Current paper: Marron, Todd and Ahn (2007) • Links to more papers: Ahn (2007) • JAVA Implementation of DWD: caBIG (2006) • SDPT3 Software: Toh (2007)

27. Batch and Source Adjustment Recall from Class Notes 8/28/07 • For Stanford Breast Cancer Data (C. Perou) • Analysis in Benito, et al (2004) Bioinformatics, 20, 105-114. https://genome.unc.edu/pubsup/dwd/ • Adjust for Source Effects • Different sources of mRNA • Adjust for Batch Effects • Arrays fabricated at different times

28. Source Batch Adj: Biological Class Col. & Symbols

29. Source Batch Adj: Source Colors

30. Source Batch Adj: PC 1-3 & DWD direction

31. Source Batch Adj: DWD Source Adjustment

32. Source Batch Adj: Source Adj’d, PCA view

33. Source Batch Adj: S. & B Adj’d, Adj’d PCA

34. Why not adjust using SVM? • Major Problem: Proj’d Distrib’al Shape Triangular Dist’ns (opposite skewed) • Does not allow sensible rigid shift

35. Why not adjust using SVM? • Nicely Fixed by DWD • Projected Dist’ns near Gaussian • Sensible to shift

36. Why not adjust by means? • DWD is complicated: value added? • Xuxin Liu example… • Key is sizes of biological subtypes • Differing ratio trips up mean • But DWD more robust (although still not perfect)

37. Why not adjust by means? Next time: Work in before and after, slides like 138-141 from DWDnormPreso.ppt In Research/Bioinf/caBIG

38. Twiddle ratios of subtypes

39. Why not adjust by means? DWD robust against non-proportional subtypes… Mathematical Statistical Question: Are there mathematics behind this? (will answer next time…)

40. DWD in Face Recognition • Face Images as Data (with M. Benito & D. Peña) • Male – Female Difference? • Discrimination Rule? • Represented as long vector of pixel gray levels • Registration is critical

41. DWD in Face Recognition, (cont.) • Registered Data • Shifts and scale • Manually chosen • To align eyes and mouth • Still large variation • See males vs. females???

42. DWD in Face Recognition , (cont.) • DWD Direction • Good separation • Images “make sense” • Garbage at ends? (extrapolation effects?)

43. DWD in Face Recognition , (cont.) • Unregistered Version • Much blurrier • Since features don’t properly line up • Nonlinear Variation • But DWD still works • Can see M-F differ’ce?

44. DWD in Face Recognition , (cont.) • Interesting summary: • Jump between means (in DWD direction) • Clear separation of Maleness vs. Femaleness

45. DWD in Face Recognition , (cont.) • Fun Comparison: • Jump between means (in SVM direction) • Also distinguishes Maleness vs. Femaleness • But not as well as DWD

46. DWD in Face Recognition , (cont.) Analysis of difference: Project onto normals • SVM has “small gap” (feels noise artifacts?) • DWD “more informative” (feels real structure?)

47. DWD in Face Recognition, (cont.) • Current Work: • Focus on “drivers”: (regions of interest) • Relation to Discr’n? • Which is “best”? • Lessons for human perception?

48. Outcomes Data Breast Cancer Study (C. M. Perou): • Outcome of interest = death or survival • Connection with gene expression? Approach: • Treat death vs. survival during study as “classes” • Find “direction that best separates the classes”

49. Outcomes Data Find “direction that best separates the classes”

50. Outcomes Data Find “direction that best separates the classes”