Tier I: Mathematical Methods of Optimization

2. Tier I: Mathematical Methods of Optimization Section 3: Nonlinear Programming

3. Introduction to Nonlinear Programming • We already talked about a basic aspect of nonlinear programming (NLP) in the Introduction Chapter when we considered unconstrained optimization.

4. Introduction to Nonlinear Programming • We optimized one-variable nonlinear functions using the 1st and 2nd derivatives. • We will use the same concept here extended to functions with more than one variable.

5. Multivariable Unconstrained Optimization • For functions with one variable, we use the 1st and 2nd derivatives. • For functions with multiple variables, we use identical information that is the gradient and the Hessian. • The gradient is the first derivative with respect to all variables whereas the Hessian is the equivalent of the second derivative

6. The Gradient • Review of the gradient (): For a function “f”, of variables x1, x2, …, xn: Example:

7. The Hessian • The Hessian (2) of f(x1, x2, …, xn) is:

8. Hessian Example • Example (from previously):

9. Unconstrained Optimization The optimization procedure for multivariable functions is: • Solve for the gradient of the function equal to zero to obtain candidate points. • Obtain the Hessian of the function and evaluate it at each of the candidate points • If the result is “positive definite” (defined later) then the point is a local minimum. • If the result is “negative definite” (defined later) then the point is a local maximum.

10. Positive/Negative Definite • A matrix is “positive definite” if all of the eigenvalues of the matrix are positive (> 0) • A matrix is “negative definite” if all of the eigenvalues of the matrix are negative (< 0)

11. Positive/Negative Semi-definite • A matrix is “positive semi-definite” if all of the eigenvalues are non-negative (≥ 0) • A matrix is “negative semi-definite” if all of the eigenvalues are non-positive (≤ 0)

12. Example Matrix Given the matrix A: The eigenvalues of A are: This matrix is negative definite

13. Unconstrained NLP Example Consider the problem: Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2 – x2x3 + (x3)2 + x3 First, we find the gradient with respect to xi:

14. Unconstrained NLP Example Next, we set the gradient equal to zero: So, we have a system of 3 equations and 3 unknowns. When we solve, we get:

15. Unconstrained NLP Example So we have only one candidate point to check. Find the Hessian:

16. Unconstrained NLP Example The eigenvalues of this matrix are: All of the eigenvalues are > 0, so the Hessian is positive definite. So, the point is a minimum

17. Unconstrained NLP Example Unlike in Linear Programming, unless we know the shape of the function being minimized or can determine whether it is convex, we cannot tell whether this point is the global minimum or if there are function values smaller than it.

18. Method of Solution • In the previous example, when we set the gradient equal to zero, we had a system of 3 linear equations & 3 unknowns. • For other problems, these equations could be nonlinear. • Thus, the problem can become trying to solve a system of nonlinear equations, which can be very difficult.

19. Method of Solution • To avoid this difficulty, NLP problems are usually solved numerically. • We will now look at examples of numerical methods used to find the optimum point for single-variable NLP problems. These and other methods may be found in any numerical methods reference.

20. Newton’s Method When solving the equation f (x) = 0 to find a minimum or maximum, one can use the iteration step: where k is the current iteration. Iteration is continued until |xk+1 – xk| <e where e is some specified tolerance.

21. Newton’s Method Diagram Newton’s Method approximates f(x) as a straight line at xkand obtains a new point (xk+1), which is used to approximate the function at the next iteration. This is carried on until the new point is sufficiently close to x*. Tangent of f(x) at xk x x* xk xk+1 f(x)

22. Newton’s Method Comments • One must ensure that f (xk+1) < f (xk) for finding a minimum and f (xk+1) > f (xk) for finding a maximum. • Disadvantages: • Both the first and second derivatives must be calculated • The initial guess is very important – if it is not close enough to the solution, the method may not converge

23. Regula-Falsi Method This method requires two points, xa & xb that bracket the solution to the equation f (x) = 0. where xc will be between xa & xb. The next interval will be xc and either xa or xb, whichever has the sign opposite of xc.

24. Regula-Falsi Diagram The Regula-Falsi method approximates the function f(x) as a straight line and interpolates to find the root. f(x) xc xa x x* xb

25. Regula-Falsi Comments • This method requires initial knowledge of two points bounding the solution • However, it does not require the calculation of the second derivative • The Regula-Falsi Method requires slightly more iterations to converge than the Newton’s Method

26. Multivariable Optimization • Now we will consider unconstrained multivariable optimization • Nearly all multivariable optimization methods do the following: • Choose a search direction dk • Minimize along that direction to find a new point: where k is the current iteration number and ak is a positive scalar called the step size.

27. The Step Size • The step size, ak, is calculated in the following way: • We want to minimize the function f(xk+1) = f(xk +akdk) where the only variable is ak because xk & dk are known. • We set and solve for ak using a single-variable solution method such as the ones shown previously.

28. Steepest Descent Method • This method is very simple – it uses the gradient (for maximization) or the negative gradient (for minimization) as the search direction: for So,

29. Steepest Descent Method • Because the gradient is the rate of change of the function at that point, using the gradient (or negative gradient) as the search direction helps reduce the number of iterations needed x2 f(x) = 5 -f(xk) f(x) = 20 f(xk) xk f(x) = 25 x1

30. Steepest Descent Method Steps So the steps of the Steepest Descent Method are: • Choose an initial point x0 • Calculate the gradient f(xk) where k is the iteration number • Calculate the search vector: • Calculate the next x:Use a single-variable optimization method to determine ak.

31. Steepest Descent Method Steps • To determine convergence, either use some given tolerance e1 and evaluate:for convergence Or, use another tolerance e2 and evaluate:for convergence

32. Convergence • These two criteria can be used for any of the multivariable optimization methods discussed here Recall: The norm of a vector, ||x|| is given by:

33. Steepest Descent Example Let’s solve the earlier problem with the Steepest Descent Method: Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2 – x2x3 + (x3)2 + x3 Let’s pick

34. Steepest Descent Example Now, we need to determine a0

35. Steepest Descent Example Now, set equal to zero and solve:

36. Steepest Descent Example So,

37. Steepest Descent Example Take the negative gradient to find the next search direction:

38. Steepest Descent Example Update the iteration formula:

39. Steepest Descent Example Insert into the original function & take the derivative so that we can find a1:

40. Steepest Descent Example Now we can set the derivative equal to zero and solve for a1:

41. Steepest Descent Example Now, calculate x2:

42. Steepest Descent Example So,

43. Steepest Descent Example Find a2: Set the derivative equal to zero and solve:

44. Steepest Descent Example Calculate x3:

45. Steepest Descent Example Find the next search direction:

46. Steepest Descent Example Find a3:

47. Steepest Descent Example So, x4 becomes:

48. Steepest Descent Example The next search direction:

49. Steepest Descent Example Find a4:

50. Steepest Descent Example Update x5: