1. Constrained Optimization Rong Jin

2. Outline • Equality constraints • Inequality constraints • Linear Programming • Quadratic Programming

3. Optimization Under Equality Constraints • Maximum Entropy Model • English ‘in’  French • {dans (1), en (2), à (3), au cours de (4), pendant (5)}

4. Reducing variables • Representing variables using only p1 and p4 • Objective function is changed • Solution: p1= 0.2, p2 = 0.3, p3 =0.1, p4 = 0.2, p5 = 0.2

5. Maximum Entropy Model for Classification • It is unlikely that we can use the previous simple approach to solve such a general • Solution: Lagrangian

6. Equality Constraints: Lagrangian • Introduce a Lagrange multiplier  for the equality constraint • Construct the Lagrangian • Necessary condition • A optimal solution for the original optimization problem has to be one of the stationary point of the Lagrangian

7. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • Stationary points

8. Lagrange Multipliers • Introducing a Lagrange multiplier for each constraint • Construct the Lagrangian for the original optimization problem

9. Original Entropy Function Constraints Lagrange Multiplier • We have more variables • p1, p2, p3, p4, p5 and, 1, 2, 3 • Necessary condition (first order condition) • A local/global optimum point for the original constrained optimization problem  a stationary point of the corresponding Lagrangian

10. Stationary Points for Lagrangian All probabilities p1, p2, p3, p4, p5 are expressed as functions of Lagrange multipliers s

11. Dual Problem • p1, p2, p3, p4, p5 are expressed as functions of s • We can even remove the variable 3 • Put together necessary condition • Still difficult to solve

12. Dual Problem • p1, p2, p3, p4, p5 are expressed as functions of s • We can even remove the variable 3 • Put together necessary condition • Still difficult to solve

13. Dual Problem • Dual problem • Substitute the expression for ps into the Lagrangian • Find the s that MINIMIZE the substituted Lagrangian

14. Expression for ps Substituted Lagrangian Dual Problem Original Lagrangian Finding s such that the above objective function is minimized

15. Dual Problem Primal Problem Dual Problem • Using dual problem • Constrained optimization  unconstrained optimization • Need to change maximization to minimization • Only valid when the original optimization problem is convex/concave (strong duality) x*=* When convex/concave

16. Maximum Entropy Model for Classification • Introduce a Lagrange multiplier for each linear constraint

17. Original Entropy Function Consistency Constraint Normalization Constraint Maximum Entropy Model for Classification • Construct the Lagrangian for the original optimization problem

18. Stationary points: first derivatives are zero Sum of conditional probabilities must be one Stationary Points Conditional Exponential Model !

19. Dual Problem

20. Dual Problem

21. Dual Problem

22. Dual Problem What is wrong?

23. Dual Problem

24. Dual Problem

25. Dual Problem

26. Dual Problem

27. Dual Problem Minimizing L  maximizing the log-likelihood

28. Support Vector Machine • Having many inequality constraints • Solving the above problem directly could be difficult • Many variables: w, b,  • Unable to use nonlinear kernel function

29. Two cases: • g(x) = c, • g(x) > c   =0 Non-negative Lagrange Multiplier Inequality Constraints: Modified Lagrangian • Introduce a Lagrange multiplier  for the inequality constraint • Construct the Lagrangian • Karush-Kuhn-Tucker (KKT) condition • A optimal solution for the original optimization problem will satisfy the following conditions

30. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KT conditions • Expressing objective function using  • Solution is =3

31. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KT conditions • Expressing objective function using  • Solution is =3

32. Expressing objective function using  • Solution is =3 Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KKT conditions

33. MinMax + SVM Model • Lagrange multipliers for inequality constraints

34. SVM Model • Lagrangian for SVM model • Karush-Kuhn-Tucker condition

35. SVM Model • Lagrangian for SVM model • Karush-Kuhn-Tucker condition

36. Dual Problem for SVM • Express w, b,  using  and 

37. Dual Problem for SVM • Express w, b,  using  and  • Finding solution satisfying KKT conditions is difficult

38. Dual Problem for SVM • Rewrite the Lagrangian function using only and  • Simplify using KT conditions

39. Maximize  Minimize Dual Problem for SVM • Final dual problem

40. Quadratic Programming Find Subject to

41. Find Subject to Linear Programming • Very very useful algorithm • 1300+ papers • 100+ books • 10+ courses • 100s of companies • Main methods • Simplex method • Interior point method Most important: how to convert a general problem into the above standard form

42. Find Subject to Example • Need to change max to min

43. Find Subject to Example • Need change  to 

44. Find Subject to Example • Need to convert the inequality

45. Find Subject to Example • Need change |x3|