Créer une présentation
Télécharger la présentation

Download

Download Presentation

15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu

179 Vues
Download Presentation

Télécharger la présentation
## 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Artificial Intelligence: Representation and Problem**SolvingOptimization (1): Optimization and Convex Optimization 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu.edu Wean Hall 4126**Logistics**• Complete the CMU 'course hours worked' survey Fei Fang**Recap**• What we have learned so far… • Search & Satisfiability in discrete space • 8-queen, Graph coloring, 3-SAT,… • Finite options • This section: Generalize to discrete / continuous space Fei Fang**Outline**• Optimization • Convex Optimization Fei Fang**Optimization Problem: Definition**• Optimization Problem: Determine value of optimization variable within feasible region/set to optimize optimization objective • Optimization variable • Feasible region/set • Optimization objective • Optimal solution: • Optimal objective value s.t. Fei Fang**Optimization Problem: Definition**• Optimization variable • Discrete variables: Combinatorial optimization • Continuous variables: Continuous optimization • Mixed: Some variables are discrete, and some are continuous s.t. Fei Fang**Optimization Problem: Definition**• Unconstrained optimization: • Constrained optimization: • Find a feasible point can already be difficult s.t. Fei Fang**Optimization Problem: Definition**• Optimization objective • : Feasibility problem • Simple functions • Linear function • Convex function (this lecture) • Complicated functions • Even can be implicitly represented through an algorithm which takes as input, and outputs a value s.t. Fei Fang**Optimization Problem: Definition**• Minimization can be converted to maximization (and vice versa) s.t. s.t. Same optimal solutionOptimal objective value Fei Fang**Optimization Problem: Example**• Example: Traveling Salesman Problem (TSP) • Problem: cities, distance from city to city is , find a tour (a closed path that visits every city exactly once) with minimal total distance • Variable : ordered list of cities being visited • is the index of the city being visited • Feasible set • Objective function s.t. Fei Fang**Optimization Problem: Example**• Example: 8-Queens Problem • Variable : location of the queen in each column • is the row index of the queen in column • Feasible set • Objective function (dummy) • Variable : index of row and column of the queen • Feasible set • ;; • Objective function (dummy) s.t. Fei Fang**Optimization Problem: Example**• Example: Linear Regression • Problem: Find such that , • Variable • Feasible region • Objective function ? Fei Fang**Optimization Problem: How to Solve**• No general way to solve • Many algorithms developed for special classes of optimization problems (i.e., when and satisfy certain constraints • Convex optimization problem (CO) • Linear Program (LP) • (Mixed) Integer Linear Program (MILP) • Quadratic program (QP), (Mixed) Integer Quadratic program (MIQP), Semidefinite program (SDP), Second-order cone program (SOCP), … • Existing solvers and code packages for these problems • Cplex (LP, MILP, QP), Gurobi (LP, MILP, MIQP), GLPK (LP, MILP), Cvxopt (CO), DSDP5 (SDP), MOSEK (QP, SOCP), Yalmip (SDP), … Fei Fang**Optimization Problem: Why Useful**• Why formulate problems as optimization problems? • For many class of optimization problems, algorithms or algorithmic frameworks have been developed • Decouple “representation” and “problem solving” • Lazy mode • Formulate a problem as an optimization problem • Identify which class the formulation belongs to • Call the corresponding solver • Done! Fei Fang**Convex Optimization: Definition**• Convex Optimization Problem: • An optimization problem whose optimization objective is a convex function and feasible region is a convex set • A special class of optimization problem s.t. Fei Fang**Convex Optimization: Definition**• Convex combination • A point between two points • Given , a convex combination of them is any point of the form where • When , is called a strict convex combination of Fei Fang**Convex Optimization: Definition**• Convex set • Any convex combination of two points in the set is also in the set • A set is convex if , , Fei Fang**Quiz 1: Convex Set**• Which of the following sets are convex? • A: • B: • C: • D: , where and are convex sets • E: Fei Fang**Convex Optimization: Definition**• Convex function • Value in the middle point is lower than average value • Let be a convex set. A function is convex in if , • If , we simply say is convex Fei Fang**Convex Optimization: Definition**• How to determine if a functions is convex? • Prove by definition • Use properties • Sum of convex functions is convex • If convex, then is convex • Convexity is preserved under a linear transformation • If , convex, then is convex • If is a twice differentiable function of one variable, is convex on an interval iff (if and only if) its second derivative in Fei Fang**Convex Optimization: Definition**• (not required) If is a twice continuously differentiable function of variables, is convex on iff its Hessian matrix of second partial derivatives is positive semidefinite on the interior of is positive semidefinite in if , is positive semidefinite in iff all eigenvalues of are non-negative Alternatively, prove Fei Fang**Convex Optimization: Definition**• Concave function • A function is concave if is convex • Let be a convex set. A function is concavein if , • The following is a convex optimization problem if is a concave function and is a convex set s.t. Fei Fang**Convex Optimization: Definition**• Affine function • An affine function is a function of the form where • If a function is both convex and concave in , then is an affine function • Proof (not required): • Let . • Clearly is convex and concave. • By definition, satisfies (i) ; (ii) . • Let be the canonical basis of , then . • Therefore where . • So Fei Fang**Quiz 2: Convex Function**• Which of the following functions are convex? • A: • B: • C: • D: • E: Counter Example for D:, , . , http://m.wolframalpha.com/input/?i=z%3D-xy Fei Fang**Convex Optimization: Local Optima=Global Optima**• Given an optimization problem, a point is globally optimal if and • Given an optimization problem, a point is locally optimal if and such that and • Theorem 1: For a convex optimization problem, all locally optimal points are globally optimal s.t. Fei Fang**Convex Optimization: Local Optima=Global Optima**• Proof of Theorem 1: Prove by contradiction. Assume is a local optima with optimality radius , and . Let where . (1) due to convexity of (2) (3) So is not local optima. Contradiction. Fei Fang**Convex Optimization: How to Solve**• Recall discrete setting • Local search • Iteratively improving an assignment • Continuous and differentiable setting • Iteratively improving value of • Based on gradient Fei Fang**Convex Optimization: How to Solve**• For , gradientis the vector of partial derivatives • A multi-variable generalization of the derivative • Point in the direction of steepest increase in Fei Fang**Convex Optimization: How to Solve**• Gradient descent: iteratively update the value of • A simple algorithm for unconstrained optimization • Variants • How to choose , e.g., • How to update , e.g., • How to define “convergence”, e.g., Fei Fang**Convex Optimization: How to Solve**• Theorem 2: For differentiable and small enough , for any with , • can be convex or non-convex • (Informal) For convex and differentiable and small enough , gradient descent converges to global optimum Proof (not required): (Recall Taylor expansion ) Consider a small , use Taylor expansion Last inequality holds for since Fei Fang**Convex Optimization: How to Solve**• Projected Gradient Descent: iteratively update the value of while ensuring • projects a point to the constraint set • Variants • How to choose , e.g., Fei Fang**Convex Optimization: How to Solve**• Unconstrained and differentiable • Gradient descent • Set derivative to be 0 • Closed form solution • (Not covered) Newton’s Method (if twice differentiable) • Constrained and differentiable • Projected gradient descent • (Not covered) Interior Point Method • (Not covered) Non-differentiable • -Subgradient Method • Cutting Plane Method Fei Fang**Convex Optimization: Apply**• Model a problem as a convex optimization problem • Define variable, feasible set, objective function • Prove it is convex (convex function + convex set) • Solve the convex optimization problem • Build up the model • Call a solver • Examples: fmincon (MATLAB), cvxpy (Python), cvxopt (Python), cvx (MATLAB) • Map the solution back to the original problem Fei Fang**Summary**Optimization Problems Convex Programs Gradient Descent Fei Fang**Convex Optimization: Additional Resources**• Text book • Convex Optimization,Chapters1-4Stephen Boyd and LievenVandenbergheCambridge University Presshttps://web.stanford.edu/~boyd/cvxbook/ • Online course • Stanford University, Convex Optimization I (EE 364A), taught by Stephen Boyd • http://ee364a.stanford.edu/courseinfo.html • https://youtu.be/McLq1hEq3UY Fei Fang**Acknowledgment**• Some slides are borrowed from previous slides made by J. Zico Kolter Fei Fang