1 / 50

10701 Recitation 5 Duality and SVM

10701 Recitation 5 Duality and SVM. Ahmed Hefny. Outline. Langrangian and Duality The Lagrangian Duality Examples Support Vector Machines Primal Formulation Dual Formulation Soft Margin and Hinge Loss. Lagrangian. Consider the problem s.t.

teryl
Télécharger la présentation

10701 Recitation 5 Duality and SVM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 10701 Recitation 5Duality and SVM Ahmed Hefny

  2. Outline • Langrangian and Duality • The Lagrangian • Duality • Examples • Support Vector Machines • Primal Formulation • Dual Formulation • Soft Margin and Hinge Loss

  3. Lagrangian • Consider the problem s.t. • Add a Lagrange multiplier for each constraint

  4. Lagrangian • Lagrangian • Setting gradient to 0 gives • [Feasible point] [Cannot decrease except by violating constraints]

  5. Lagrangian • Consider the problem s.t. • Add a Lagrange multiplier for each constraint

  6. Duality

  7. Duality • Primal problem s.t. • Equivalent to

  8. Duality • Primal problem s.t. • Equivalent to

  9. Duality • Dual Problem • Dual function: • Concave, regardless of the convexity of the primal • Lower bound on primal Lagrangian Dual Function

  10. Duality Primal Problem

  11. Duality Primal Problem • For each row (choice of ), • pick the largest element • then select the minimum.

  12. Duality Dual Problem • For each column (choice of ), • pick the smallest element • then select the maximum.

  13. Duality Claim:

  14. Duality Claim: For any The difference between primal minimum And dual maximum is called duality gap duality gap = 0  Strong Duality

  15. Duality When does

  16. Duality When does is a saddle point

  17. Duality When does is a saddle point Necessity  By definition of dual Sufficiency 

  18. Duality When does is a saddle point Necessity  By definition of dual Sufficiency  The dual at is the upper bound

  19. Duality • If strong duality holds, KKT conditions apply to optimal point • Stationary Point • Primal Feasibility • Dual Feasibility () • Complementary Slackness () • KKT conditions are • Sufficient • Necessary under strong duality

  20. Example: LP • Primal s.t.

  21. Example: LP • Primal s.t. • Lagrangian

  22. Example: LP • Dual Function

  23. Example: LP • Dual Function • Set gradient w.r.t to 0

  24. Example: LP • Dual Function • Set gradient w.r.t to 0 • Dual Problem s.t. Why keep this as a constraint ?

  25. Example: LASSO • We will use duality to transform LASSO into a QP

  26. Example: LASSO Primal What is the dual function in this case ?

  27. Example: LASSO Reformulated Primal s.t. Dual

  28. Example: LASSO Dual Setting gradient to zero gives

  29. Example: LASSO • Dual Problem s.t.

  30. Support Vector Machines docs.opencv.org

  31. Support Vector Machines • Find the maximum margin hyper-plane • “Distance” from a point to the hyper-plane is given by • Max Margin:

  32. Support Vector Machines • Max Margin • Unpleasant (max min ?) • No Unique Solution

  33. Support Vector Machines • Max Margin s.t. ???

  34. Support Vector Machines • Max Margin s.t.

  35. Support Vector Machines • Max Margin s.t.

  36. Support Vector Machines • Max Margin (Canonical Representation) s.t. • QP, much better than

  37. SVM Dual Problem Recall that the Lagrangian is formed by adding a Lagrange multiplier for each constraint.

  38. SVM Dual Problem Fix and minimize w.r.t :

  39. SVM Dual Problem Fix and minimize w.r.t : Plug-in Constraint (why ?)

  40. SVM Dual Problem Dual Problem s.t. Another QP. So what ?

  41. SVM Dual Problem • Only Inner products  Kernel Trick • Complementary Slackness  Support Vectors • KKT conditions lead to Efficient optimization algorithms (compared to general QP solver)

  42. SVM Dual Problem • Classification of a test point • To get use the fact that for any support vector. • For numerical stability, average over all support vectors.

  43. Soft Margin SVM Hard Margin SVM , where

  44. Soft Margin SVM Hard Margin SVM , where loss regularization

  45. Soft Margin SVM Relax it a little bit , where

  46. Soft Margin SVM Relax it a little bit , where

  47. Soft Margin SVM Relax it a little bit

  48. Soft Margin SVM Equivalent Formulation s.t.

  49. Conclusions • Duality allows for establishing a lower bound on minimization problem. • Key idea • “min max” upper bounds “max min” • Strong Duality  Necessity of KKT Conditions • Duality on SVMs • Kernel Trick • Support Vectors • Soft Margin SVM = Hinge Loss

  50. Resources • Bishop, “Pattern Recognition and Machine Learning”, Chp 7 • Gordon & Tibshirani, 10725 Optimization (Fall 2012) Lecture Slides: http://www.cs.cmu.edu/~ggordon/10725-F12/schedule.html • Fiterau, Kernels and SVM “http://alex.smola.org/teaching/cmu2013-10-701/slides/6_Recitation_Kernels.pdf”

More Related