Introduction to Optimization

1. Introduction to Optimization Anjela Govan North Carolina State University SAMSI NDHS Undergraduate workshop 2006

2. What is Optimization? • Optimization is the mathematical discipline which is concerned with finding the maxima and minima of functions, possibly subject to constraints.

3. Where would we use optimization? • Architecture • Nutrition • Electrical circuits • Economics • Transportation • etc.

4. What do we optimize? • A real function of n variables • with or without constrains

5. Unconstrained optimization

6. Optimization with constraints

7. Lets Optimize • Suppose we want to find the minimum of the function

8. Review max-min for R2 • What is special about a local max or a local min of a function f (x)? at local max or local min f’(x)=0 f”(x) > 0 if local min f”(x) < 0 if local max

9. Review max-min for R3

10. Review max-min for R3 • Second Derivative Test • Local min, local max, saddle point • Gradient of f – vector (df/dx, df/dy, df/dz) direction of fastest increase of f • Global min/max vs. local min/max

11. Gradient Descent Method Examples • Minimize function • Minimize function

12. Gradient Descent Method Examples • Use function gd(alpha,x0) • Does gd.m converge to a local min? Is there a difference if  > 0 vs.  < 0? • How many iterations does it take to converge to a local min? How do starting points x0 affect number of iterations? • Use function gd2(x0) • Does gd2.m converge to a local min? • How do starting points x0 affect number of iterations and the location of a local minimum?

13. How good are the optimization methods? • Starting point • Convergence to global min/max. • Classes of nice optimization problems Example: f(x,y) = 0.5(x2+y2),  > 0 Every local min is global min.

14. Other optimization methods • Non smooth, non differentiable surfaces  can not compute the gradient of f  can not use Gradient Method • Nelder-Mead Method • Others

15. Convex Hull • A set C is convex if every point on the line segment connecting x and y is in C. • The convex hull for a set of points X is the minimal convex set containing X.

16. Simplex • A simplex or n-simplex is the convex hull of a set of (n +1) . A simplex is an n-dimensional analogue of a triangle. • Example: • a 1-simplex is a line segment • a 2-simplex is a triangle • a 3-simplex is a tetrahedron • a 4-simplex is a pentatope

17. Nelder-Mead Method • n = number of variables, n+1 points • form simplex using these points; convex hull • move in direction away from the worst of these points: reflect, expand, contract, shrink • Example: • 2 variables  3 points  simplex is triangle • 3 variables  4 points  simplex is tetrahedron

18. Nelder-Mead Method – reflect, expand

19. Nelder-Mead Method-reflect, contract

20. A tour of Matlab: Snapshots from the minimizationAfter 0 steps

21. A tour of Matlab: Snapshots from the minimizationAfter 1 steps

22. A tour of Matlab: Snapshots from the minimizationAfter 2 steps

23. A tour of Matlab: Snapshots from the minimizationAfter 3 steps

24. A tour of Matlab: Snapshots from the minimizationAfter 7 steps

25. A tour of Matlab: Snapshots from the minimizationAfter 12 steps

26. A tour of Matlab: Snapshots from the minimizationAfter 30 steps (converged)

27. fminsearch function • parameters: q =[C,K] • cost function: • Minimize cost function [q,cost]= fninsearch(@cost_beam, q0,[],time,y_tilde)

28. Our optimization problem • In our problem • Our function: • cost function “lives” in R3 • 2 parameters C and K, n=2 • Simplex is a triangle

29. Done!