1 / 18

Understanding Kernel PCA and the Perceptron Algorithm in High-Dimensional Spaces

This document explores the fundamentals of Kernel PCA and the Perceptron algorithm, focusing on their roles in machine learning, particularly with high-dimensional data. It discusses the dual representation, kernel trick, and challenges like overfitting in high-dimensional spaces. The document covers essential concepts such as dimensionality reduction, eigenvectors, and the significance of choosing the right kernel function. Additionally, it highlights the differences between PCA and LDA, employing practical examples like face recognition to illustrate their applications.

adriel
Télécharger la présentation

Understanding Kernel PCA and the Perceptron Algorithm in High-Dimensional Spaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Is PCA enough? Irena Váňová

  2. . . . A1 A2 A3 Dot product B

  3. for i=1...N if then end end * * * * * o o * * o * o * o o Perceptron algorithm … labels of classes repeat until no sample is misclassified

  4. Find the coefficients is equivalent to find Gram matrix Rewitten algorithm – dual form repeat for i=1...N if then end end until no sample is misclassified • In the dual representation, the data points only appear inside dot products • Many algorithms have dual form

  5. Mapping to higher dimensions • Perceptron works for linear separable problems • There is a computational problem (very large vectors) • Kernel trick:

  6. Example of kernels • Polynomial kernels • Gaussian kernels • Infinit dimensions • Separated by a hyperplane • Good kernel? • Bad kernel! • Almost diagonal

  7. repeat for i=1...N if then end end until no sample ismisclassified Kernel Perceptron • We precompute • We are in implicitly in higher dimensions (too high?) • Generalization problem - easy to overfit in high dimensional spaces

  8. Kernel trick • Kernel function • Use: replacing dot products with kernels • Implicit mapping to feature space • Solve the computational problem • Can make it possible to use infinite dimensions • Conditons: continuous, symmetric, positive definite • Information ‘bottleneck’: contains all necessaryinformation for the learning algorithm • Fuses information about the data AND the kernel

  9. PCA • Orthogonal linear transformation • The greatest variance = first coordinate, … • Rotation around mean value • Dimensionality reduction • many dimensions = high correlation

  10. Singular value decomposition • W,T – unitary matrix ( ) • Columns of W,V? • Basis vector, eigenvectors of XTX, resp. XXT n n m n m m n

  11. PCA • Data with zero mean, SVD n m n m m m covariance matrix (1/n) eigenvectors

  12. Kernel PCA • Projections of data onto few larger eigenvectors equation for PCA kernel function equation for high-dim. PCA We don’t know eigenvector explicitly - only vector of numbers which identify the vector projection onto k-th eigenvector

  13. KPCA example

  14. If something does wrong PCA is blind

  15. LDA • fundamental assumption: normal distribution • First: same covariance matrix , full rank

  16. LDA • fundamental assumption: normal distribution • First: only full rank • Kernel variant

  17. Example LDA • Face recognition – eigenfaces

  18. LDA versus PCA

More Related