Algorithm Design Paradigms

1. Algorithm Design Paradigms • Reduce to known problem • Divide and conquer, partitioning. • Dynamic programming. • Greedy method. • Backtracking. • Branch and Bound. • Recursion. • Approximations. • Geometric methods. • Integer programming. • Probabilistic techniques.

2. Reduce to Known Problem Example 1: Determine if an array of n numbers contains repeated elements. Solution 1: Compare each element to every other element. O(n2) Solution 2: Sort (e.g., by mergesort) the n numbers. Then determine if there is a repeat in O(n) steps. Total: (n log n) steps!

3. Another Example • Given a list of n points in the plane, determines if any 3 of them are collinear (lie on the same line). Solution 1: Using a triple loop, compare all distinct triples of points, so this takes O(n3) time. Solution 2: O(n2log n). For each point P in the list do for each point Q in the list do compute the slope of the line connecting P and Q and save it in a list determine (Example 1) if there are any duplicate

4. Divide and Conquer • Divide the problem into a number of subproblems • There must be base case (to stop recursion). • Conquer (solve) each subproblem recursively • Combine (merge) solutions to subproblems into a solution to the original problem

5. Example: Find the MAX and MIN • Obvious strategy (Needs 2n - 3 compares) • Find MAX (n - 1 compares) • Find MIN of the remaining elements (n - 2) • Nontrivial strategy • Split the array in half • Find the MAX and MIN of both halves • Compare the 2 MAXes and compare the 2 MINs to get overall MAX and MIN. • In this example, we only consider the number of comparisons and ignore all other operations.

6. Procedure mm(i, j: integer; var lo, hi: integer); • begin • if i = j • thenbegin lo := a[i]; hi := a[i] end • else if i = j-1 • thenbegin if a[i] < a[j] • thenbegin lo := a [i]; hi := a[j] end • elsebegin lo := a [j]; hi := a[i] end • end • elsebegin • m := (i+j) div 2; • mm(i, m, min1, max1); • mm(m+1, j, min2, max2); • lo := MIN(min1, min2); • hi := MAX(max1, max2) • end • end.

7. Analysis • Solving the above, we have T(n) = 3n/2 –2 if n is a power of 2. • This can be shown to be a lower bound.

8. Note • More accurately, it should be

9. Balancing • It is generally best to divide problems into subproblems of roughly EQUAL size. Very roughly, you want binary search instead of linear search. Less roughly, if in the MAX-MIN problem, we divide the set by putting one element in subset 1 and the rest in subset 2, the execution tree would look like: 2(n – 1) + 1 = 2n –3 Comparisons would Be needed.

10. Mergesort • Obvious way to sort n numbers (selection sort): • Find the minimum (or maximum) • Sort the remaining numbers recursively • Analysis: Requires (n-1) + (n-2) + ... + 1 = n(n-1)/2 =Q(n2) comparisons. • Clearly this method is not using the balancing heuristics.

11. A Divide and Conquer Solution • The following requires onlyQ(n log n) comparisons. Mergesort • Divide: divide the given n-element array A into 2 subarrays of n/2 elements each • Conquer: recursively sort the two subarrays • Combine: merge 2 sorted subarrays into 1 sorted array • Analysis: T(n) =Q(1) + 2T(n/2) +Q(n) =Q(n log n)

12. The Pseudocode • Procedure mergesort(i, j: integer); • var m: integer; • begin • if i < j then • begin • m := (i+j) div 2; • mergesort(i, m); • mergesort(m+1, j); • merge(i, m, j) • end; • end.

13. Illustration

14. The Merging Process • merge(x, y, z: integer) uses another array for temporary storage. • merge segments of size m and n takes m + n – 1 compares in the worst case.

15. smallest smallest Merging auxiliary array

16. Merging smallest smallest auxiliary array

17. Merging

18. Merging

19. Merging

20. Merging

21. Merging first half exhausted second half exhausted

22. input output Summary function D&C(P: problem): solution; if size(P) is small enough then S = solve(P) else divide P into P1, P2, P3, …, Pk; S1=D&C(P1); S2=D&C(P2); …, Sk=D&C( Pk); S = merge(S1, S2, S3, …, Sk); return(S);

23. Summary • Divide and Conquer with Balancing is a useful technique if the subproblems to be solved are independent (no redundant computations). • Also the dividing and merging phases must be efficient. • The Fibonacci problem was an example where the subproblems were not independent. • Usually, either dividing or merging the subproblems will be trivial. • Problem is usually, but not always, divide into two parts. • Divide into ONE part: Binary search. • Divide into > 2 parts: Critical path problem.

24. Two Dimensional Search • You are given an m  n matrix of numbers A, sorted in increasing order within rows and within columns. Assume m = O(n). Design an algorithm that finds the location of an arbitrary value x, in the matrix or report that the item is not present. Is your algorithm optimal? • How about probe the middle of the matrix? • It seems we can eliminate 1/4 data with one comparison and it yields 3 subproblems of size about 1/4 of the original problem • Is this approach optimal? What is the recurrence in this case? • T(n) = 3T(n/4) + O(1) • ????

25. Well, It is Wrong! • T(n) = 3T(n/4) + O(1) is not correct, because the subproblems are of size n/2. • The correct recurrence for the solution is T(n) = 3T(n/2) + O(1)  T(n) = O( )

26. > > < < = Illustration of Idea

27. A Q(n) algorithm • c = n, r = 1 • if c = 1 or r = m then use binary search to locate x. • compare x and A[r, c]: • x = A[r, c] -- report item found in position (r, c). • x > A[r, c] -- r = r + 1; goto step 2 • x < A[r, c] -- c = c - 1; goto step 2. At most m + n comparisons are required.

28. Selection • The Problem: Given a sequence S of n elements and an integer k, determine the kth smallest element in S. • Special cases: • k = 1, or k = n:Q(n) time needed. • k = n/2 : trivial method -- O(n2) steps • sort then select -- O(nlog n) steps • Lower bound:W(n).

29. A Linear Time Algorithm procedure SELECT(S, k) 1. if |S| < Q then sort S and return the kth element durectly else subdivide S into |S|/Q subsequence of Q elements (with up to Q-1 leftover elements). 2. Sort each subsequence and determine its median.   3. Call SELECT recursively to find m, the median of the |S|/Q medians found in setp 2.   4. Create three subsequences L, E, and G of elements of S smaller than, equal to, and larger than m, respectively. 5. if |L| ≥ k then call SELECT(L, k) else if |L| + |E| ≥ k then return(m) else SELECT(G, k - |L| - |E|).

30. Analysis of Selection • Let t(n) be the running time of SELECT. Step 1 Step 2 Step 3 Step 4 Step 5 O(n) O(n) t(n/Q) O(n) t(3n/4) Q

31. The Complexity t(n) = t(n/Q) + t(3n/4) + O(n) = t(n/5) + t(3n/4) + O(n) Take Q = 5 Since 1/5 + 3/4 < 1, we have t(n) = Q(n) . • Recall that the solution of the recurrence relation t(n) = t(pn) + t(qn) + cn, when 0 < p + q < 1, is Q(n)

32. Multiplying Two n Bit Numbers • Here we are using the log-cost model, counting bits. • The naive pencil-and-paper algorithm: • This uses n2 multiplications, (n-1)2 additions (+ carries). In fact, this is also divide and conquer.

33. Karatsuba's algorithm, 1962 :O(n1.59 ) • Let X and Y each contain n bits. Write X = a b and Y = c d where a; b; c; d are n/2 bit numbers. Then XY = (a2n/2 + b)(cn/2 + d) = ac2n + (ad + bc)2n/2 + bd • This breaks the problem up into 4 subproblems of size n/2, which doesn't do us any good. Instead, Karatsuba observed that XY = (2n +2n/2)ac + 2n/2 (a-b)(d-c) + (2n/2 + 1)bd = ac2n +ac2n/2 + 2n/2 (a-b)(d-c) + bd2n/2 +bd = ac2n + (ad + bc)2n/2 + bd

34. Polynomial multiplication • Straightforward multiplication: O(n2). • Using D&C approach: O(n1.59) • Using FFT technique: O(nlog n)

35. A D&C Approach

36. A Modified D&C Solution : O(n1.59 ) • Any idea for further improvement?

37. Matrix Multiplication

38. Complexity (on uniprocessor) • Best known lower bound:W(n2) (assume m = Q(n) and k = Q(n) ) • Straightforward algorithm: O(n3). • Strassen's algorithm: O(nlog 7) = O(n2.81). • Best known sequential algorithm: O(n2.376) ? • The best algorithm for this problem is still open.

39. The Straightforward Method • It takes O(mnk) = O(n3) time.

40. A D&P approach

41. Strassen's algorithm • T(n) = 7T(n/2) +O(n2) = O(nlog 7) = O(n2.81)

42. Quicksort • Quicksort is a simple divide-and-conquer sorting algorithm that practically outperforms Heapsort. • In order to sort A[p..r] do the following: • Divide: rearrange the elements and generate two subarrays A[p..q] and A[q+1..r] so that every element in A[p..q] is at most every element in A[q+1..r]; • Conquer: recursively sort the two subarrays; • Combine: nothing special is necessary. • In order to partition, choose u = A[p] as a pivot, and move everything < u to the left and everything > u to the right.

43. Quicksort • Although mergesort is O(n log n), it is quite inconvenient for implementation with arrays, since we need space to merge. • In practice, the fastest sorting algorithm is Quicksort, which uses partitioning as its main idea.

44. Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

45. swap me Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

46. swap me Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

47. swap me Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

48. swap me swap me Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

49. Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross

50. swap me Partitioning in Quicksort • How do we partition the array efficiently? • choose partition element to be rightmost element • scan from right for smaller element • scan from left for larger element • exchange • repeat until pointers cross