1. Sparse Kernel Machines

2. Introduction • Kernel function k(xn,xm) must be evaluated for all pairs of training points • Advantageous (computationally) to look at kernel algorithms that have sparse solutions • new predictions depend only on kernel function evaluated at a few training points • possible solution: support vector machines (SVM)

3. Maximum Margin Classifiers (1) • Setup: • y(x) = w’φ(x) + b, φ(x) = feature-space transformation Training data comprises • N input data vectors x1,...,xN • N target vectors t1,...,tNtn∈{-1,1} • new samples classified according to sign of y(x) • assume linearly separable data for the moment • y(x)>0 for points of class +1 (i.e. tn = +1) • y(x)<0 for points having tn = -1 • therefore,tny(xn) > 0 for all training points

4. Maximum Margin Classifiers (2) • There may be many solutions to separate the data SVM tries to minimize the generalization error by introducing the concept of a margin i.e. smallest distance between the decision boundary and any of the samples • Decision boundary is chosen to be that for which the margin is maximized • decision boundary thus only dependent on points closest to it.

5. Maximum Margin Classifiers (3)

6. If we consider only points that are correctly classified, then we have tny(xn) > 0 for all n • The distance of xn to the decision surface is then • tny(xn) • w = tn(wTφ(xn) + b) • w • ● Optimize w and b to maximize this distance => solve • (7.2) • (7.3) • ● Quite hard to solve => need to get around it

7. Maximum Margin Classifiers (4)

15. 7.1.1 Overlapping class distributions

18. An alternative, equivalent formulation of the support vector machine, known as the ν-SVM

19. the parameter ν, which replaces C, can beinterpreted as both an upper bound on the fraction of margin errors (points for whichξn > 0 and hence which lie on the wrong side of the margin boundary and which mayor may not be misclassified) and a lower bound on the fraction of support vectors.

21. 7.1.2 Relation to logistic regression

22. 7.1.3 Multi-class SVM

23. 7.1.4 SVMs for regression

24. 7.1.4 SVMs for regression

25. 7.1.4 SVMs for regression

26. 7.1.4 SVMs for regression

27. 7.1.4 SVMs for regression &7.1.5 Computational learning theory

28. Relevance Vector Machines (1)

29. Relevance Vector Machines (2)7.2.1 RVM for regression

30. Relevance Vector Machines (3)7.2.1 RVM for regression

31. Relevance Vector Machines (4)7.2.1 RVM for regression

32. Relevance Vector Machines (5)7.2.1 RVM for regression

33. Relevance Vector Machines (6)7.2.1 RVM for regression

34. Relevance Vector Machines (8)7.2.2 Analysis of sparsity

36. Relevance Vector Machines (9)7.2.3 RVM for classification

37. Relevance Vector Machines (10)7.2.3 RVM for classification

38. Relevance Vector Machines (11)7.2.3 RVM for classification

39. Relevance Vector Machines (12)7.2.3 RVM for classification