Optimization for Nonlinear P rogramming

1. Optimization for Nonlinear Programming By Liyun Zhang Aug.9th,2012

2. Method for Single Variable Unconstraint Optimization : 1. Quadratic Interpolation method 2. Golden Section method • Method for Multivariable Constraint Optimization Method 1. Lagrange Multiplier Method 2. Exterior-point Penalty Method Topic

3. Solve for: unconstraint optimization problems with single variable and unimodal function • Advantage and Disadvantage: converges fast but UNRILIABLE (you will see it in a later example) Quadratic Interpolation

4. Mathematical Background: target function: f(x) select three points x0, x1, x2 on f(x) such that: x0<x1<x2, f2=min(f0, f1, f2) there must exist a quadratic function: q(x)=a0+a1x+a2x^2 which connects these points such that: q(x0)=a0+a1x0+a2x0^2=f(x0) q(x1)=a0+a1x1+a2x1^2=f(x1) q(x2)=a0+a1x2+a2x2^2=f(x2) `

5. suppose (x3,f3) is the minimum point of q(x), then q’(x)=0, or x3= - we can solve that a1= a2= - so x3= Mathematical Background(cont.):

6. Algorithm: Initialization: Determine x0, x1 and x2 which is known to contain the minimum of the function f(x). Step 1 If x3-x1<ε (a sufficiently small number), then the minimum occurs at xo=x3 and stop iteration, else go to Step 2. Step 2 Evaluate x3 and x1 If x3<x1, go to Step 3, else go to Step 4. Step 3 Evaluate f(x3) and f(x1) If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x0 x2=x1 x1=x3 If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x3 x1=x1 x2=x2

7. Step 4 Evaluate f(x3) and f(x1) If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x1 x1=x3 x2=x2 If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x0 x1=x1 x2=x3 Step 5 Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3. Algorithm(cont.): illustration code

8. Example 1. Find the minimum point and value of the function f(x)=(x^2-2)^2/2-1 Solution: f1001=inline('(x.*x-2).^2/2-1','x'); a=0; b=5; TolX=1e-5; TolFun=1e-8; MaxIter=100; [xoq,foq]=Opt_Quadratic(f1001,[a,b],TolX,TolFun,MaxIter) xoq = 1.414187140007763 foq = -0.999999997207487 Application and Comparison:

9. Fminbndmethod attempts to solve the optimum of one variable function within a fixed interval. The most common syntaxes are: [x, f]=fminbnd (fun, x1, x2) [x, f]=fminbnd (fun, x1, x2, options) In Ex. 1, we have x1=0 x2=5 and the result is: x = 1.414214286263162 f = -0.999999999997904

10. Application and Comparison(cont.):

11. Golden Section Method • Solve for: unconstraint optimization problems with single variable and unimodal function • Advantages: 1. more reliable than quadratic interpolation method 2. still converges fast since: a. rely on interval approximation instead of derivative calculations. b. the rate of the convergence is fixed to avoid the wider interval being used many times which may slow down the rate of convergence. c. interior points are selected to reduce the bounds as quickly as possible.

12. Select the intermediate points x3 and x4: c/a=a/b c/(b-c)=a/b so k=a/b=(1+ )/2 x3=x1+1/(k+1)*(x2-x1) x4=x1+k/(k+1)*(x2-x1) Mathematical Background:

13. Initialization: Determine x1 and x2 which is known to contain the minimum of the function f(x). Step 1 Determine two intermediate points x3 and x4 such that x3=x1+(1-r)*h x4=x1+r*h where r=k/(k+1)= ( -1)/2 h=x2-x1 Step 2 If x4-x3<ε (a sufficiently small number), then the minimum occurs at xo=x3 (f(x3) < f(x4)) or xo=x4 (f(x3) > f(x4)) and stop iterating, else go to Step3. Step3 Evaluate f(x3) and f(x4) If f(x3)<f(x4), then determine new x1 and x2 as shown following, and go to Step4. x1=x1 x2=x4 If f(x3)>f(x4), then determine the new x1 and x2 as shown following, and go to Step4. x1=x3 x2=x2 Step 4 Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3, x4. Algorithm: illustration code

14. Application and Comparison Example 2. Find the minimum point and value of the function x-(x^2-2)^3/2, x∈[0,4] Solution: f1002=inline('x-(x.*x-2).^3/2','x'); a=0; b=4; TolX=1e-4; TolFun=1e-4; MaxIter=100;   [xog,fog]=Opt_Golden(f1002,a,b,TolX,TolFun,MaxIter) xog = 3.999999954235401 fog = -1.367999892407431e+003 [xoq,foq]=Opt_Quadratic(f1002,[a,b],TolX,TolFun,MaxIter) xoq = 1.938216986786059 foq = -0.772297597277559

15. Application and Comparison(cont.):

16. Apply primarily on: optimization problems with multivariable functions and equality constraints • Mathematical Backgrounds: Consider the function z=f(x,y) which is subject to ϕ(x,y)=0 or y=g(x), suppose x0 is the minimum point of z=f(x,g(x)), then x0 is the solution of the following equation system: f’x+ λ ϕ’x=0 f’y+ λ ϕ’y=0 ϕ(x,y)=0 More generally, we can introuduce a transformed function L(x1,x2…xm; λ1,λ2…λn)=f(x1,x2…xm)+ ∑λnϕn(x1,x2…xn) and find all x satisfying the following equation system f’xi+ ∑ λnϕn’xi=0 (i=1,2…m) ϕk(x1,x2…xm) =0 (k=1,2…n) substitute {x} back into f(x1,x2…xm) and then compare their value Lagrange Multiplier Method

17. Example 3. Find the minimum point and value of the function f(x1,x2)=x1+x2, s.t x1^2+x2^2=2 Solution: code >> [xo,yo,fo]=Lag() xo = -1 1 yo = -1 1 fo = -2 2 so minimum is -2 at (-1,-1) Algorithm and comparison

18. Apply primarily on: optimization problems with multivariable functions and equality or inequality constraints, and the initial guess point is out of the feasible domain. • Advantage: work well when the initial guess point is out of the domain so that the computation is subjected to less restrictions compared with the interior- point penalty method. Exterior-point Penalty Method

19. Consider the function f(x) which is subject to a series of equality or inequality constraints Ci(x)>=0(i=1,2…m),to find the minimum point of f(x), we combine these conditions into a single function: F(x)=f(x)+c*∑g(Ci(x)). penalty coefficients: ci=c1*p^(i-1) where p>=2 penalty function: g(Ci(x))=min(0, Ci(x))^2 Note that the optimum point of the unconstrained function will eventually converge to the original constrained one when c*∑g(Ci(x))<=ɛ. One way to search for the optimum point of the unconstrained function is the Newton method, which resorts to the derivative method. Mathematical Background

20. Initialization Set up the initial optimum guess point x0, c1,p(p>=2), accuracy ɛ>0, and let k=1. Step 1 Combine the given conditions into a new function F(x)=f(x)+ci*(∑hp(x)^2+∑gs(x)^2), hp(x)=0 (p=1,2…m) gs(x)>=0 (s=1,2…n) Step 2 Search for the minimum of F(x) from the point xk-1 with unconstraint optimization methods. Step 3 Let S(xk)= ck* (∑hp(x)^2+∑gs(x)^2), if S(xk)<= ɛ, stop iteration, else let k=k+1 and go to step 2 Algorithm: code

21. Example 4. Find the minimum point and value of the function f(x1,x2)=x1^2-x1*x2+x2-x1+1, s.t 2x1+3x2-9=0 x2^2+x1^2-6>=0 x1>=0 x2>=0 Solution: let c1=0.05, p=2, (x1,x2)=(2,2) >> syms x1 x2; >> f=x1^2-x1*x2+x2-x1+1; >> g=[x1^2+x2^2-6;x1;x2]; h=[2*x1+3*x2-9]; >> [x,minf]=minEPF(f,[2 2],g,h,0.05,2,[x1 x2]) >> x= 1.4000 2.0668 minf= 0.7333 Application and Comparison

22. Fminconattempts to solve constraint optimization problems of the form: min f(x) subject to: A*x<=b, Aeq*x=beq (linear constraints) x subject to: c(x)<=0, ceq(x)=0 (nonlinear constraints) lb<=x<=ub(bounds) The common syntax for fmincon is [x,fval]=fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon) where x0 is the initial guess. In Ex. 4, we have >> A=[-1 0; 0 -1]; >> b=[0; 0]; >> Aeq=[2 3]; >> beq=[9]; and the result is x = 1.399999998247254 2.066666667835164 fval= 0.733333333333334