Greedy Algorithms

Greedy Algorithms Greed is good. (Some of the time)

Outline • Elements of greedy algorithm • Greedy choice property • Optimal substructures • Minimum spanning tree • Kruskal’s algorithm • Prim’s algorithm • Huffman code • Activity selection Chapter 3: Greedy Algorithms

Introduction • Concepts • Choosing the best possible choice at each step. • This decision leads to the best over all solution. • Greedy algorithms do not always yield optimal solutions. Chapter 3: Greedy Algorithms

Elements of Greedy Algorithms

Greedy-choice property • A globally optimal solution is derived from a locally optimal (greedy) choice. • When choices are considered, the choice that looks best in the current problem is chosen, without considering results from subproblems. Chapter 3: Greedy Algorithms

Optimal substructures • A problem has optimal substructure if an optimal solution to the problem is composed of optimal solutions to subproblems. • This property is important for both greedy algorithms and dynamic programming. Chapter 3: Greedy Algorithms

Dynamic programming A choice is made at each step. The choice made at each step usually depends on the solutions to subproblems. Dynamic-programming problems are often solved in a bottom-up manner. Greedy algorithm The best choice is made at each step and after that the subproblem is solved. The choice made by a greedy algorithm may depend on choices so far, but it cannot depend on any future choices or on the solutions to subproblems. A greedy strategy usually progresses in a top-down fashion, making one greedy choice after another, reducing each given problem instance to a smaller one. Greedy Algorithm v.s. Dynamic Programming Chapter 3: Greedy Algorithms

Steps in Design Greedy Algorithms • Determine the optimal substructure of the problem. • Develop a recursive solution. • Prove that at any stage of the recursion, one of the optimal choices is the greedy choice. Thus, it is always safe to make the greedy choice. • Show that all but one of the subproblems induced by having made the greedy choice are empty. • Develop a recursive algorithm that implements the greedy strategy. • Convert the recursive algorithm to an iterative algorithm. Chapter 3: Greedy Algorithms

Shortcuts Design • Form the optimization problem so that, after a choice is made and there is only one subproblem left to be solved. • Prove that there is always an optimal solution to the original problem that makes the greedy choice, so that the greedy choice is always safe. • Demonstrate that, having made the greedy choice, what remains is a subproblem with the property that if we combine an optimal solution to the subproblem with the greedy choice we have made, we arrive at an optimal solution to the original problem. Chapter 3: Greedy Algorithms

Minimum Spanning Tree

Definitions • Let G = (V, E)be an undirected graph. T is a minimum spanning tree of G if T ⊆ E is an acyclic subset that connects all of the vertices and whose total weight w(T )=  w(u, v) is minimized. (u,v)∈T • Let A be a subset of some minimum spanning tree. An edge (u, v) is a safe edge for A if A{(u, v)} is also a subset of a minimum spanning tree. Chapter 3: Greedy Algorithms

Definitions • Let G = (V, E)be an undirected graph. A cut (S, V − S)of G is a partition of V. An edge (u, v)E crosses the cut (S, V − S)if one of its endpoints is in S and the other is in V − S. A cut respects a set A of edges if no edge in A crosses the cut. An edge is a light edge crossing a cut if its weight is the minimum of any edge crossing the cut. Chapter 3: Greedy Algorithms

Theorem Let G = (V, E)be a connected, undirected graph with a real-valued weight function w defined on E, A be a subset of E that is included in some minimum spanning tree for G, (S, V − S)be any cut of G that respects A. If (u, v)is a light edge crossing (S, V − S), thenedge (u, v)is safe for A. S A u x light edge v y V-S Chapter 3: Greedy Algorithms

Proof Let T be a minimum spanning tree that includes A. If (u,v) is in T, then (u,v) is safe for A. If (u,v) is not in T, then T contains an edge (x,y) crossing S and V-S. Since both (u,v) and (x,y) cross S and V-S, there is a cycle for edges in T  (u,v). Another spanning tree T’ can be created from T. S A u x light edge T v y V-S Chapter 3: Greedy Algorithms

S A u x light edge T’ v y V-S Proof S A u x light edge T v y V-S Remove edge (x,y) -> cut the cycle Add edge (u,v) -> create a cycle T’ is created from T. Chapter 3: Greedy Algorithms

Proof Thus,T’is a spanning tree that includes A. Next, we need to show that T’is a minimum spanning tree. From the construction of T’, T’= T-{(x,y)}  {(u,v)}. Thus, w(T’ ) = w(T) - w(x,y) +w(u,v). Since (u,v) is a light edge cross S and V-S, w(u,v)  w(x,y). Thus, w(T’ ) w(T). Since T is a minimum spanning tree, w(T’ ) = w(T). Then, T’isalso a minimum spanning tree . As a result, (u, v)is safe for A. Chapter 3: Greedy Algorithms

Corollary Let G = (V, E)be a connected, undirected graph with a real-valued weight function w defined on E, A be a subset of E that is included in some minimum spanning tree for G, and C = (VC, EC)be a connected component (tree) in the forest GA= (V, A). If (u, v)is a light edge connecting C to another tree in GA, then (u, v)is safe for A. Proof (VC, V−VC)respects A, and (u, v)is a light edge for this cut. Therefore, (u, v)is safe for A. Chapter 3: Greedy Algorithms

Kruskal’s Algorithm • Concept • Build a forest of minimum spanning trees. • Repeatedly connect the trees to create a subset of a minimum spanning tree, until all nodes are covered. • In connecting two trees, choose the edge of the least weight. • Let C1 and C2 denote the two trees that are connected by (u, v). Since (u, v)must be a light edge connecting C1 to some other tree, we need to prove that (u, v)is a safe edge for C1. • Kruskal’s algorithm is a greedy algorithm, because at each step it adds to the forest an edge of least possible weight. Chapter 3: Greedy Algorithms

Example of Kruskal’s Algorithm 12 8 1 9 2 9 5 3 11 7 5 8 3 1 10 Chapter 3: Greedy Algorithms

Kruskal’s Algorithm MST-KRUSKAL(G,w) A =  for each vertex v in V [G] do MAKE-SET(v) sort the edges of E in nondecreasing order by weight w for each edge (u, v)in E, taken in nondecreasing order by weight do if FIND-SET(u) FIND-SET(v) then A = A  {(u, v)} UNION(u, v) return A Chapter 3: Greedy Algorithms

Kruskal’s Algorithm: Complexity MST-KRUSKAL(G,w) A =  for each vertex v in V [G] do MAKE-SET(v) sort the edges of E in nondecreasing order by weight w for each edge (u, v)in E, taken in nondecreasing order by weight do if FIND-SET(u) FIND-SET(v) then A = A  {(u, v)} UNION(u, v) return A O(v) O(e lg e) O((v+e) lg v) e  v2; O(e lg v) Chapter 3: Greedy Algorithms

Prim’s Algorithm • Prim’s algorithm has the property that the edges in the set A always form a single tree. • The tree starts from an arbitrary root vertex r. • Grow the tree until it spans all the vertices inV. • At each step, a light edge is added to the tree A that connects A to an isolated vertex of GA =(V, A). Chapter 3: Greedy Algorithms

Example of Prim’s Algorithm 12 8 1 9 2 9 5 3 11 7 5 8 3 1 10 Chapter 3: Greedy Algorithms

Prim’s Algorithm PRIM(G,w, r) for each u in V[G] do key[u] = ∞ π[u] = NIL key[r] = 0 Q = V[G] while Q  do u = EXTRACT-MIN(Q) for each v in Adj[u] do if v in Q and w(u, v) < key[v] then π[v] = u key[v] = w(u, v) Chapter 3: Greedy Algorithms

Execution of Prim’s Algorithm 12 8 0 9 12 3 8 1 9 2 9 2 1 3 5 3 11 7 5 8 3 5 7 1 10 5 1 10 Chapter 3: Greedy Algorithms

Prim’s Algorithm:Complexity PRIM(G,w, r) for each u in V[G] do key[u] = ∞ π[u] = NIL key[r] = 0 Q = V[G] while Q  do u = EXTRACT-MIN(Q) for each v in Adj[u] do if v in Q and w(u, v) < key[v] then π[v] = u key[v] = w(u, v) Also build min-heap O(v) O(e lg v) O(lg v) O(v lg v) O(lg v) Chapter 3: Greedy Algorithms

Huffman Code

Problem • Find an optimal prefix code representing a set of characters in a file, where each character has a frequency of occurrences. • Prefix code • Codes in which no codeword is also a prefix of some other codeword. • {0, 110, 101, 111, 1000, 1001} is a prefix code. • {0, 110, 101, 111, 1010, 0111} is not prefix code • Optimality • The code yields a file with the minimum number of bits. Chapter 3: Greedy Algorithms

Creating Huffman Code: Example 100 1 64 0 1 0 30 34 1 15 0 1 0 1 0 ฉ: 5 ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 0 100 101 110 1110 1111 Chapter 3: Greedy Algorithms

Optimal Code • An optimal code for a file is always represented by a full binary tree, in which every nonleaf node has two children. • If C is the alphabet from which the characters are drawn and all character frequencies are positive, then the tree for an optimal prefix code has exactly |C| leaves, one for each letter of the alphabet, and exactly |C|−1 internal nodes. Chapter 3: Greedy Algorithms

Tree for 1 letter Tree for 2 letters Tree for 4 letters A 1 1 2 B Tree for 3 letters C A B 4 3 1 2 3 A C 1 2 B 4 A 3 A 1 B 1 2 2 3 Full Binary Trees for Prefix Code Chapter 3: Greedy Algorithms

Tree for 5 letters Tree for 4 letters B B C A C A 4 3 1 2 D 3 1 2 5 4 C D B 4 5 B 3 A C A 1 2 4 3 1 2 Full Binary Trees for Prefix Code Chapter 3: Greedy Algorithms

Creating Huffman Code: Example กขคงจฉ: 100 1 ขคงจฉ: 64 0 1 0 งจฉ: 30 ขค: 34 1 จฉ: 15 0 1 0 1 0 ฉ: 5 ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 100 101 110 1110 1111 0 Chapter 3: Greedy Algorithms

Theorem • A full binary tree for an optimal prefix code for C letters has exactly C leaves, and exactly C−1 internal nodes. Proof by induction. Basis: C=1. If there is one letter, the binary tree requires only 1 leaf node, and 0 internal node. Induction hypotheses: For C< n, the full binary tree for an optimal prefix code for C letters has exactly C leaves, and exactly C−1 internal nodes. Chapter 3: Greedy Algorithms

Theorem • Induction Step: Let T be a full binary tree for an optimal prefix code for C+1 letters. To create a full binary tree or optimal prefix code, we can take a full binary tree for an optimal prefix code for C letters, and add another leaf node L by either • adding a new root node R and put L and the old root of T as its children, or • replacing a leaf node N of T by an internal node and put L and N as its children. In either case, the number of leaf nodes of T is C+1 and the number of internal nodes is C. Chapter 3: Greedy Algorithms

Creating Huffman Code: Algorithms HUFFMAN(C) n = |C| Q = C for i = 1 to n − 1 do allocate a new node z z.left = x = EXTRACT-MIN(Q) z.right = y = EXTRACT-MIN(Q) z.f =x.f + y.f INSERT(Q, z) ►Return the root of the tree. return EXTRACT-MIN(Q) Use min-heap for Q O(n) to build min-heap O(n lg n) O(lg n) Chapter 3: Greedy Algorithms

Greedy-choice property of Huffman Code Let C be an alphabet in which each character c C has frequency f [c], and x and y be two characters in C having the lowest frequencies. • Then, there exists an optimal prefix code for C in which the codewords for x and y have the same length and differ only in the last bit. Chapter 3: Greedy Algorithms

Proof • The idea of the proof is to : • take the tree T representing an arbitrary optimal prefix code • modify it to make a tree representing another optimal prefix code such that the characters x and y appear as sibling leaves of maximum depth in the new tree. • If we can do this, then their codewords will have the same length and differ only in the last bit. Chapter 3: Greedy Algorithms

Proof Let a and b be two characters that are sibling leaves of maximum depth in T . Assume that f [a] ≤ f [b] and f [x] ≤ f [y]. Since f [x] and f [y] are the two lowest leaf frequencies, in order, and f [a] and f [b] are two arbitrary frequencies, in order, we have f [x] ≤ f [a] and f [y] ≤ f [b]. Chapter 3: Greedy Algorithms

x y a b a y x b Proof Exchange the positions of a and x in T to produce T’ . B(T) − B(T’ ) =  f(c) dT (c)−  f(c)dT’ (c) cCcC = f [x] dT (x)+ f [a] dT (a)− f [x] dT’ (x)− f [a] dT’ (a) = f [x] dT (x) + f [a] dT (a)− f [x] dT (a) − f [a] dT (x) = ( f [a] − f [x])( dT (a)− dT (x)) ≥ 0 T ’ T Chapter 3: Greedy Algorithms

a y x b Proof Then, exchange the positions of b and y in T’to produce T’’. Similarly, itdoes not increase the cost, and so B(T’ )− B(T’’ ) 0. Therefore, B(T’’ )≤ B(T). Since T is optimal, B(T)≤ B(T’’ ). Then, B(T’’ )= B(T). Thus, T’’is an optimal tree in which x and y appear as sibling leaves of maximum depth. T ’ T’’ a b x y Chapter 3: Greedy Algorithms

Optimal-substructure Property Let C be a given alphabet with frequency f [c] defined for each character c C, x and y be two characters in C with minimum frequency, C’ be the alphabet C with characters x, y removed and (new) character z added, so that C’ = C − {x, y}  {z}; define f for C’ as for C, except that f [z] = f [x] + f [y], and T be any tree representing an optimal prefix code for the alphabet C ’. • Then the tree T , obtained from T ’ by replacing the leaf node for z with an internal node having x and y as children, represents an optimal prefix code for the alphabet C. Chapter 3: Greedy Algorithms

Proof Show that the cost B(T)can be expressed in terms of the cost B(T’)by considering the component costs. For each c C − {x, y}, we have • dT(c)= dT ’ (c). • f [c] dT (c)= f [c] dT ’ (c). Since dT(x)= dT(y)= dT ’(z)+ 1, we have f [x]dT(x)+ f [y]dT(y)= ( f [x] + f [y])( dT’ (z)+ 1) = f [z] dT ’(z)+ ( f [x] + f [y]) Thus, B(T)= B(T ’)+ f [x] + f [y]. That is, B(T ’)= B(T)− f [x] − f [y] . Chapter 3: Greedy Algorithms

Proof We now prove by contradiction. Suppose T does not represent an optimal prefix code for C. Then there exists a tree T ’’ such that B(T ’’) < B(T). Without loss of generality, T ’’ has x and y as siblings. Chapter 3: Greedy Algorithms

Proof Let T ’’’ be the tree T ’’ with the common parent of x and y replaced by a leaf z with frequency f [z] = f [x] + f [y]. Then, B(T ’’’)= B(T ’’)− f [x] − f [y] < B(T)− f [x] − f [y] = B(T ’). We reach a contradiction to the assumption that T ’ represents an optimal prefix code for C’. Thus, T must represent an optimal prefix code for the alphabet C. Chapter 3: Greedy Algorithms

Interval Scheduling

Problem definition • Let S be {a1, a2, . . . , an} of n proposed activities that wish to use the same resource. • Only one activity can use the resource at a time. • Each activity aihas a start time siand a finish time fi, where 0 ≤ si < fi < ∞. • If selected, activity aitakes place during the half-open time interval [si , fi). • Activities aiand ajare compatible if the intervals [si , fi)and [s j , f j)do not overlap . • The activity-selection problem is to select a largest subset of mutually compatible activities. Chapter 3: Greedy Algorithms

A1 A2 A3 Example S Chapter 3: Greedy Algorithms

Subproblems • Si j= {akS : fi≤ sk < fk≤ s j} • the subset of activities in S that can start after activity aifinishes and finish before activity ajstarts. • Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞. • Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1. Chapter 3: Greedy Algorithms

ak Optimal Solution • Let Ai jbean optimal solution to Si j . • Let c[i, j ] be the number of activities in a maximum-size subset of mutually compatible activities in Si j. • c[i, j ] = 0 for i ≥ j (i.e. Si j= ) • c[i, j ] = c[i, k] + c[k, j ] + 1 if ak be an activity in Ai j. Then, • c[i, j ] =  0 if Si j=  max {c[i, k] + c[k, j ] + 1} if Si j i<k<j akSi j Chapter 3: Greedy Algorithms

Greedy Algorithms