The greedy method

1. The greedy method Chapter 3

2. 3-0 The Greedy Method • Suppose that a problem can be solved by a sequence of decisions. • The greedy method has that each decision is locally optimal. These locally optimal solutions will finally add up toa globally optimal solution. • Only a few optimization problems can be solved by the greedy method.

3. 3-0 The Greedy Method • Step by step to construct a “feasible” solution. • At each step, “ select ” the best feasible candidate. • Useful to construct heuristic or approximation algorithms. (Chap 9) • Example: • Dijkstra’s Single Source • Kruskal’s Minimum Spanning Tree

4. 3-0 The Greedy Method: Easy example • E.g. Pick k numbers out of n numbers such that the sum of these k numbers is the largest. • Algorithm: FOR i = 1 to k pick out the largest number and delete this number from the input. ENDFOR

5. 3-0 The Greedy Method • E.g. Find a shortest path from v0 to v3. • The greedy method can solve this problem. • The shortest path: 1 + 2 + 4 = 7.

6. 3-0 The Greedy Method • E.g. Find a shortest path from v0 to v3 in the multi-stage graph. • Greedy method: v0v1,2v2,1v3 = 23 • Optimal: v0v1,1v2,2v3 = 7 • The greedy method does not work.

7. 3-0 The Greedy Method • How to solve? • dmin(i,j): minimum distance between i and j. • This problem can be solved by the dynamic programming method (Chap 7).

8. 3-0 The Greedy Method • E.g. How to win in a game? • The greedy method does not work because it never looks ahead.

9. 3-1 Minimal Spanning Trees • It may be defined on Euclidean space points or on a graph. • G = (V, E): weighted connected undirected graph • Spanning tree : S = (V, T), T  E, undirected tree • Minimal spanning tree(MST) : a spanning tree with the smallest total weight.

10. 3-1 Minimal Spanning Trees • A graph and one of its minimum costs spanning tree

11. 3-1 Construct MST

12. 3-1 Kruskal’s algorithm for finding MST • Step 1: Sort all edges • Step 2: Add the next smallest weight edge to the forest if it will not cause a cycle (By the SET and UNION method). • Step 3: Stop if n-1 edges. Otherwise, go to Step2.

13. 3-1 Construct MST • Time complexity: O(|E| log|E|) • Time Complexity: • Sort the edges of G : O(|E|log|E|) • Union two disjoint trees (sets): |V|-1 times (union operation) • Find an element in a set (Check whether an edge can be added or not ): 2|E| times (find operation) (To understand the above two operations, see Chapter 9 or [CLR90] T. Cormen , C.Leiserson and R. Rivest, “Introduction to Algorithms”, MIT press 1990.) • The totally running time = O(|E|log|E|) O(|E|)

14. 3-1 Construct MST • How do we check if a cycle is formed when a new edge is added? • By the SET and UNION method. • A tree in the forest is used to represent a SET. • If (u, v)  E and u, v are in the same set, then the addition of (u, v) will form a cycle. • If (u, v) E and uS1 , vS2 , then perform UNION of S1 and S2 .

15. Extra: SETS AND DISJOINT SET UNION • 有 1, 2, …, n 這麼多個元素，分成任意個集合，任兩個集合的交集都是空集合。需要作兩個 operations： • Disjoint set union. 將任兩個 sets 合成一個 set。 • Find(i).找出某個元素屬於哪個集合(或者說、判斷某兩個元素是否屬於同一個集合)。 • 例如下圖中的 7 屬於 S1，而 10 屬於 S2，4 與 6 屬於同一個 set S3 。 • 直接用 root 的名字當作集合的名字。

16. Extra: Union and Find Operations 1 5 3 5 1 7 8 9 5 2 10 1 4 2 6 10 7 8 9 2 10 7 8 9 S2 S1 S3 Note: down-up pointer 陣列 表示 5 1 S1 ∪ S2

17. 3-1 Correctness • The Correctness of Kruskal’s algorithm (Proof) • Let e1<e2<...<em be the set of edges where m=|E|. • Let T be a spanning tree produced by Kruskal’s algorithm and T’ be the minimum spanning tree of G. • We shall prove that T = T’. First, assume that TT’. • 假設T’內有一最小weight edge ei滿足eiT’但eiT. • 因為ekT’ and T, for 1k i-1,因為ei T’, T’形成一cycle, 故T’非spanning tree. 一個矛盾產生. 故T = T’. • Local Optimal  Global Optimal ei

18. 3-2 Prim’s algorithm for finding MST • Step 1: x  V, Let A = {x}, B = V - {x} • Step 2: Select (u, v)  E, u  A, v  B  (u, v) has the smallest weight between A and B • Step 3: (u, v) is in the tree. A = A  {v}, B = B - {v} • Step 4: If B = , stop; otherwise, go to Step 2. • Time complexity : O(n2), n = |V|.

19. 3.2 The Prim’s Method (Conti.) • Time Complexity = O(|V|2) • Each time, we check whether a vertex v in Y can be added to T. That is O(|V|) time. And we must repeat the checking process at |V| times. Totally, the running time is O(|V|2).

20. 3.2 The Prim’s Method (Conti.) • The Correctness of Prim’s algorithm (Proof) • Let G=(V, E) denote a weighted graph, and T be the minimum spanning tree of G . • Let T1 denote a subtree of T . Let V1 denote the set of vertices in T1 and V2=V-V1. • Let (a,b) be a minimal weighted edge in E such that aV1 and bV2. • We shall show that (a,b) must be in T. • Assume otherwise (i.e. (a,b) not in T), then there must be a path in T from a to b because a tree is connected. • Let (c,d) be an edge in that path such that cV1 and dV2. • The weight of (c,d) must be larger than the weight of (a,b). • We may therefore create another smaller spanning tree by deleting (c,d) and adding (a,b). • Thus, this show that T must not be a minimal spanning tree. • Therefore, (a,b) must be in T and Prim’s algorithm is correct.

21. 3-3 The single-source shortest path problem • Graph and shortest paths from v0 to all destinations

22. 3-3 Dijkstra’s algorithm • E.g. Cost adjacency matrix. All entries not shown are +.

23. 3-3 Dijkstra’s algorithm Action of SHORTEST_PATHS • Time complexity : O(n2)

24. 3-3 Dijkstra’s algorithm • Method • Construct a set SV such that the shortest path from v0 to each vertex S lies wholly in S. • Algorithm S = {v0}; for i = 1 to n do /*|V|=n+1*/ If (v0, vi) E then L(vi) = cost(v0, vi) else L(vi) = ; for i = 1 to n do Choose u from V-S such that L(u) is the smallest; S = S {u}; /*put u into S*/ for all w in V-S do L(w) = min{L(w), L(u)+cost(u, w)};

25. 3-3 Dijkstra’s algorithm • Time complexity : O(n2) • Analysis: • It is easy to obtain O(n2) due to the repeated operations to calculate L(w) • The lower bound to solve the single source shortest path problem is (|E|) because every edge has to be examined. In the worst case, (|E|) = (n2) thus Dijkstra’s algorithm is optimal.

26. 3-3 Dijkstra’s algorithm • Can we use Dijkstra’s algorithm to find thelongest path from a starting vertex to an ending vertex in an acyclic directed graph?

27. 3-3 Dijkstra’s algorithm • There are 3 possible ways to apply Dijkstra’s algorithm: • Convert all positive weights to be negative. Then find the shortest path. (Remark: ∞) • Give a very large positive number M. If the weight of an edge is w, now M-w is used to replace w. Then find the shortest path. (Remark: ∞) • Directly use “max” operations in stead of “min” operations. (Remark: Cycle) • All these 3 possible ways would not work! • Why? => Thinking (This may be a Mid-term test)

28. 3-3 Dijkstra’s algorithm : Other issues • The longest path(critical path) problem can be solved by the critical path method(CPM). • Step 1. Find a topological ordering. • (Activity on vertex , AOV network) • Step 2. Find the critical path. (AOE network) (see [Horiwitz 1995].) • [[Horowitz 1995] E. Howowitz, S. Sahni and D. Metha, Fundamentals of Data Structures in C++, Computer Science Press, New York, 1995

29. Topological ordering: AOV network 1 1 f b 3 0 1 1 2 g i a d e 1 c h 2 • a directed graph G • the vertices represent tasks • the edges represent precedence relations • a topological order : method • listing a vertex in the network that has no predecessor • if no such a vertex => contains a directed cycles • delete this vertex and all edges leading from it Order: abcdefghi

30. AOV Networks (Note Algorithm) • implementation • decide whether a vertex has any predecessors • maintain a count field • the number of immediate predecessor(in-degree) of each vertex • delete a vertex together with all its incident edges • represent the network by a adjacency lists • decrease the count of all vertices on its adjacency list • if count = 0 put the vertex onto a stack (the vertices with a zero count) the stack is constructed by using the count field

31. AOE Networks • Activity-on-Edge(AOE) Networks • consists of • vertices: events • signals the completion of certain activities • directed edges: the tasks to be performed (leaving a vertex v) • cannot be started until the event at vertex v has occurred • evaluation:critical path • the least amount of time to complete the project • the length of the longest path from the start vertex to the finish vertex • which activities should be speeded to reduce project length?

32. AOE Networks: Implementation Definitions:  1.etv ：earliest time of vertex (State or event)  2.ltv ：latest time of vertex (State or event)  3.ete ：earliest time of edge (Task)  4.lte ：latest time of edge (Task)

33. B E e1=3 e4=10 e9=3 e5=5 e7=8 A D G e2=4 e3=6 e8=6 C e6=4 F e10=7 Example: Building a house

34. Vertex A B C D E F G etv 0 3 6 10 13 16 23 B E e1=3 e4=10 e9=3 e5=5 e7=8 A D G e2=4 ltv 0 5 6 10 20 16 23 e3=6 e8=6 C e6=4 F e10=7 • According to topological order, compute the earliest time of vertex ( etv) => A B CD E F G, or A B C D E F G • According to inversed topological order, compute the latest time of vertex ( ltv) ※ etv Example: the in-degree of vertex is 3 (B,A,C)，thus etv(D) = max { etv(B)+5, etv(A)+4, etv(C)+4 } = 10 ※ ltv example: the out-degree of vertex is2 (G,F)，thus ltv(D) = min { ltv(G)-8, ltv(F)-6 } = 10

35. Task (Edge) e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 B E e1=3 e4=10 e9=3 ete 0 0 0 3 3 6 10 10 13 16 e5=5 e7=8 A D G e2=4 e3=6 e8=6 lte 2 6 0 10 5 6 15 10 20 16 C e6=4 F e10=7 3. According to etv and ltv of each vertex, compute ete (earliest time of edge) and lte (latest time of edge) Critical path: If ete and lte of a vertex is the same, then this vertex is an edge in critical path. e3, e6, e8, e10 ( A -> C -> D -> F -> G )

36. 3-4 The 2-way merging problem (Recall Ch2) • The number of comparisons required for the linear 2-way merge algorithm is m+n-1where m and n are the lengths of the two sorted lists respectively.

37. 3-4 The 2-way merging problem • The problem: There are m sorted list, each of length ni. What is the optimal sequence of merging process to merge these m lists into one sorted list ? Extended Binary Tree Representing a 2-way Merge

38. 3-4 The 2-way merging problem • Algorithm Input: k sorted lists Output: an optimal 2-way merge tree Method: Step 1: Generate k trees, each tree has a node with weight ni(the length of the list); (sorted k weights) => O(n log n) Step 2: Choose two trees T1 and T2 with minimum weights; Step 3: Create a new tree T whose root has T1 and T2 as its subtree, and the weight of T is the sum of weights of T1 and T2; Step 4: Replace T1 and T2 by T; Step 5: If there is only one tree left then stop; else goto Step 2.

39. 3-4 The 2-way merging problem • Time complexity of the greedy algorithm to generate an optimal extended binary tree: O(n log n) • Straight method: • Step 2: O(1) • Step 3: O(1) • Step 4: O(log n) (Insert a new node into sorted list) (It can be viewed as binary search) • Step 5: O(n) • Another method:

40. 3-4 The 2-way merging problem

41. 3-4 The 2-way merging problem • E.g. There are 6 sorted lists with lengths 2, 3, 5, 7, 11 and 13.

42. 3-4 The 2-way merging problem

43. 3-4 Huffman codes • In telecommunication, how do we represent a set of messages, each with an access frequency, by a sequence of 0’s and 1’s? • To minimize the transmission and decoding costs, we may use short strings to represent more frequently used messages. • This problem can by solved by using an extended binary tree which is used in the 2-way merging problem.

44. E.g. symbols: A, B, C, D, E, F, G freq. : 2, 3, 5, 8, 13, 15, 18 Huffman codes: A: 10100 B: 10101 C: 1011 D: 100 E: 00 F: 01 G: 11 3-4 Huffman codes A Huffman code Tree

45. 3-5 The minimal cycle basis problem • 3 cycles: A1 = {ab, bc, ca} A2 = {ac, cd, da} A3 = {ab, bc, cd, da} where A3 = A1 A2 (A  B = (AB)-(AB)) A2 = A1 A3 A1 = A2 A3 Cycle basis: {A1, A2} or {A1, A3} or {A2, A3}

46. 3-5 The minimal cycle basis problem • Def: A cycle basis of a graph is a set of cycles such that every cycle in the graph can be generated by applying (XOR) on some cycles of this basis. • Minimal cycle basis: smallest total weight of all edges in this cycle. • E.g. {A1, A2}

47. 3-5 The minimal cycle basis problem • Algorithm for finding a minimal cycle basis: • Step 1: Determine the size of the minimal cycle basis, demoted as k. • *Step 2: Find all of the cycles. Sort all cycles( by weight). • Step 3: Add cycles to the cycle basis one by one. Check if the added cycle is already combination of some cycles already existing in the basis. If it is, delete this cycle. • Step 4: Stop if the cycle basis has k cycles.

48. 3-5 The minimal cycle basis problem – detail description • Step 1 : A cycle basis corresponds to the fundamental set of cycles with respect to a spanning tree. a graph a spanning tree a fundamental set of cycles # of cycles in a cycle basis : = k = |E| - (|V|- 1) = |E| - |V| + 1

49. 3-5 The minimal cycle basis problem – detail description • Step 2: How to find all cycles in a graph ? [Reingold, Nievergelt and Deo 1977] How many cycles in a graph in the worst case? In a complete digraph of n vertices and n×(n-1) edges: Two edges form a cycle

50. 3-5 The minimal cycle basis problem – detail description • Step 3: How to check if a cycle is a linear combination of some cycles? Using Gaussian elimination.