Optimization on Graphs

1. Optimization on Graphs

2. Optimization on Graphs Edge Constraints For all Graph pairwise interactions on for ( ) Objective

3. Optimization on Graphs Edge Constraints For all Graph pairwise interactions on for ( ) ( ) Objective

4. Optimization on Graphs: An Example

5. Isotonic Regression Predict child’s height from mother’s height Model? Height of child Height of mother

6. Isotonic Regression Predict child’s height from mother’s height Model? Increasing function? Height of child Height of mother

7. Isotonic Regression Predict child’s height from mother’s height Model? Increasing function? Height of child Height of mother

8. Isotonic Regression Height of child Height of mother

9. Isotonic Regression Height of child Height of mother

10. Isotonic Regression Constraint: for all Cost: Height of child Height of mother

11. Isotonic Regression Constraint: for all Cost: Height of child Height of mother

12. Isotonic Regression Constraint: for all Cost: Height of child Height of mother

13. Isotonic Regression Constraint: for all Cost: Height of child Height of mother

14. Isotonic Regression Constraint: for all Cost: Height of child Height of mother

15. Isotonic Regression Constraint Graph Age of child Height of mother Height of child Height of mother Height of child

16. Isotonic Regression Constraint Graph Age of child Height of mother Height of mother Taller mother AND older child

17. Isotonic Regression Constraint Graph Constraint: for all Age of child Cost: Height of mother

18. Isotonic Regression Constraint Graph Constraint: for all Age of child Cost: Height of mother Height of child

19. Optimization on Graphs Graph Functions & constraints on pairs : predicted height of child Constraint: for all Cost:

20. Optimization on Graphs Graph Functions & constraints on pairs (+ barrier trick) Unconstrained problem with modified objective

21. Optimization on Graphs: Fast Algorithms

22. Optimization Primer

23. Optimization Primer Second Order Methods – “Newton Steps” 2nd order 1st order

24. Optimization Primer Second Order Methods – “Newton Steps”

25. Optimization Primer Second Order Methods – “Newton Steps”

26. Optimization Primer Second Order Methods – “Newton Steps”

27. Optimization Primer Second Order Methods – “Newton Steps”

28. Optimization Primer Second Order Methods – “Newton Steps”

29. Optimization Primer Second Order Methods – “Newton Steps”

30. Optimization Primer Second Order Methods – “Newton Steps” A good update step!

31. Second Order Methods Usually finding step that solves linear equation is too expensive! But for optimization on graphs, we can find the Newton step much faster than we can solve general linear equations

32. Graphs and Hessian Linear Equations Newton Step: find s.t. Gaussian Elimination: time for matrix . “Faster” methods: time Hessian from sum of convex functions on two variables Symmetric M-matrix Laplacian Spielman-Teng ’04: Laplacian linear equations can be solved in time

33. Graphs and Hessian Linear Equations Newton Step: find s.t. Gaussian Elimination: time for matrix . “Faster” methods: time Hessian from sum of convex functions on two variables Symmetric M-matrix Laplacian Daitch-Spielman ’08: Symmetric M-matrix linear equations can be solved in time

34. Convex Functions

35. Convex Functions No negative eigenvalues!

36. Hessians & Graphs Graph-Structured Cost Function

37. Second Derivatives 2-by-2 PSD non-negative eigenvalues If every term looks like the sum is an M-matrix

38. Second Derivatives

39. Second Derivatives

40. Second Derivatives

41. Second Derivatives

42. Second Derivatives

43. Second Derivatives

44. Second Derivatives

45. Second Derivatives

46. Second Derivatives

47. Second Derivatives

48. Second Derivatives

49. Second Derivatives 2-by-2 PSD non-negative eigenvalues If every term looks like the sum is an M-matrix

50. Second Derivatives 2-by-2 PSD non-negative eigenvalues If every term looks like Newton Step can be computed in nearly linear time!