1. CSE 3101 Prof. Andy Mirzaian Summations & Recurrence Relations

2. STUDY MATERIAL: • [CLRS] Appendix A, chapter 4 • Lecture Note 3

3. Summations: Recurrence Relations:

4. TOPICS Summations & Recurrence Relations arise in the running time analysis of algorithms. • SUMMATIONS: • Classic Methods: Arithmetic, Geometric, Harmonic, Bounded Tail • Approximating Summation by Integration • Summation by Parts • ASSUME:ASymptoticSUmmationMade Easy • RECURRENCE RELATIONS: • Iteration method • Recursion tree method • Divide-&-Conquer Recurrence: The Master Method • Guess-&-Verify method • Full History Recurrences • Variable Substitution method

5. SUMMATION

6. Classic Methods • Arithmetic (or polynomial):S(n) = Q(n f(n)) f(n) = n:1 + 2 + ··· + n = n(n+1)/2 = Q( n2 )f(n) = n2:12 + 22 + ··· + n2 = n(n+1)(2n+1)/6 = Q( n3 )f(n) = nd:1d + 2d + ··· + nd = Q( nd+1 ), for any constant real d > -1 • Geometric (or exponential):S(n) = Q(f(n)) f(n) = xn:1 +x + x2 + ··· + xn= (xn+1 –1)/(x –1) for any constant real 0|x|1= Q(xn) if x > 1 (Geometric increasing)f(n) = 2n:1+ 2 + 22 + ··· + 2n= Q(2n) (with x = 2) • Harmonic:S(n) = Q(log n) f(n) = 1/n:H(n) = 1 + 1/2 + 1/3 + ··· + 1/n lnn = Q(log n) • Bounded Tail:S(n) = Q(1) 1 + x + x2 + ··· + xn = (1 – xn+1)/(1-x) = Q(1) if 0<x<1. (Geometric decreasing)f(n) = 2-n:1 + 1/2 + 1/4 + ··· + 1/2n = Q(1) (with x = ½)f(n) = n-2:1 + 1/22 + 1/ 32 + ··· + 1/n2 = Q(1) (Arithmetic decreasing)

7. f : f monotonically non-decreasing f(x) x m-1 m m+1 m+2 . . . n n+1 Approximating by

8. f : f monotonically non-increasing f(x) x m-1 m m+1 m+2 . . . n n+1 NOTE: direction of inequalities changed Approximating by

9. Approximating by S(n) = f(1) + f(2) + … + f(n) (m = 1) Example 1: S(n) = 13 + 23 + … + n3 . f(x) = x3 : f  and f(x) dx =  x3dx = x4 / 4. n4 / 4  S(n)  ( (n+1)4 – 1)/ 4  S(n) = Q(n4) Example 2:H(n) = 1 + 1/2 + 1/3 + … + 1/n . f(x) = 1/x : f  and f(x) dx = dx /x = ln x. 1 + ln n  H(n)  ln (n+1)  H(n) = Q(log n) Caution against dividing by zero!Separate f(1) from the summation.

10.  n·n  O(n2) n/2 n/2  W(n2) Summation by Parts: Arithmetic f(n) = n: S(n) = f(1) + f(2) + f(3) + ··· + f(n). S(n) = 1 + 2 + 3 + ·······································································+ n + (n/2 –1) + n/2 + (n/2 +1) + (n/2 + 2) + • n/2 · n/2  W(n2)  0  S(n) = Q(n2)

11.  1+ log n  O(log n) 1 <1 < 4(1/4) =1 < 8(1/8) = 1 < 2(1/2)=1 >2(1/4) = ½ 1 ½ > 4(1/8) = ½ > 8(1/16) = ½ • 1 + ½ log n  W(log n) Summation by Parts: Harmonic  H(n) = Q(log n)

12. ASSUME :ASymptoticSUmmationMade Easy f: N + : S(n) = f(1) + f(2) + f(3) + ··· + f(n) maxn(f) = max { f(1) , f(2) , … , f(n) }minn(f) = min { f(1) , f(2) , … , f(n) } aven(f) = average {f(1) , f(2), … , f(n) } S(n) = n ·aven(f) 0  minn(f)  aven(f)  maxn(f) f  minn(f) = f(1) , maxn(f) = f(n)f  minn(f) = f(n) , maxn(f) = f(1) max{n ·minn(f), maxn(f) } S(n) n ·maxn(f)  S(n) = Q( g(n) · maxn(f))for some 1  g(n)  n ASSUME: A method for finding g(n) (and hence, S(n)).

13. ASSUME on monotone f(n) f: N + S(n) = f(1) + f(2) + f(3) + ··· + f(n) FACT: f is monotone  exactly one of the following holds: (1) f(n) = 0 n (2) f(n) = Q(1) (3) f(n) = w(1) and f (4) f(n) = o(1) and fand f(1) > 0 Cases (1) & (2): g(n) = Q(n), S(n) = Q(g(n) f(n)) = Q(n f(n)) Case (3): Considered next. Case (4): Similar methods apply. Exercise.

14. S(n) = f(1) + ··· + f(n) f f(n) f(m)  cf(n)  f(m+1) f(n)g(n) 0 <c < 1 m n g(n) ASSUME for f & f(n) = w(1) Step 1) Find m such that log f(m) = log f(n) – Q(1) Step 2) g(n) n – m [ Note: 1  g(n)  n ] THEOREM:f & g  S(n) = Q(f(n)g(n)).

15. ASSUME: examples Polynomial: f(n) = nd (const. d  0) : log f(m) = log f(n) – Q(1) d log m = d log n – d = d log n/2 m = n/2  g(n) = n–m = n/2. f & g S(n) = Q(f(n)g(n)) = Q(nd+1). Exponential:f(n) = 2n : log f(m) = log f(n) – Q(1) m = n – 1 g(n) = n–m = 1 f & g S(n) = Q(f(n)g(n)) = Q(2n).

16. log(1 x) = Q( ln(1 – x) ) = – Q(x) for x = o(1) Taylor Series Expansion : f (x) = ln(1– x) : Example:

17. ASSUME: examples Super-polynomial sub-exponential 1:f(n) = : log f(m) = log f(n) – Q(1) = – 1 [Now square both sides] m = (– 1)2 = n – 2+ 1  g(n) = n–m = 2–1 = Q(). f & g S(n) = Q(f(n)g(n)) = Q(). Super-polynomial sub-exponential 2:f(n) = 2n/log n : log f(m) = log f(n) – Q(1) (*)m/log m = n/log n – Q(1) [Multiply across by log m  log n ] m = n – Q(log n) [Verify this satisfies (*) with “=” ] m/log m = (n – Q(log n)) / log (n – Q(log n)) = ( n – Q(log n) ) / (log n)(1 – Q(1/n)) [See previous page] = ( n/log n – Q(1) ) / (1 – Q(1/n)) = n/log n – Q(1) [Why? Multiply sides by 1 – Q(1/n)]  g(n) = n – m = Q(log n) f & g S(n) = Q(f(n)g(n)) = Q(2n/log nlog n).

18. f f(n) f(m)  cf(n)  f(m+1) f(m+1)g(n) m m+1 n g(n) ASSUME: Lower Bound Proof THEOREM:f & g  S(n) = f(1) + ··· + f(n) = Q(f(n)g(n)). Lower bound: S(n)  S(n) – S(m) = f(m+1) + f(m+2) + ··· + f(n)  shaded rectangular area since f = f(m+1) g(n)  c f(n) g(n) W( f(n) g(n) ).

19. f f(n0) cf(n0)  f(n1) g(n0)f(n0) c2f(n0) cf(n1)  f(n2) g(n1)f(n1) g(n2)f(n2) g(n2) g(n1) g(n0) n3 n2 n1 n0 =n ASSUME: Upper Bound Proof THEOREM:f & g  S(n) = f(1) + ··· + f(n) = Q(f(n)g(n)). Upper bound: S(n)  sum of shaded rectangular areas since f = g(n0)f(n0) + g(n1)f(n1) + g(n2)f(n2) + ···  g(n) [f(n0) + f(n1) + f(n2) + ··· ] since g  g(n)f(n) [1 + c + c2 + ··· ] f(ni+1) c f(ni)  g(n)f(n) / (1 – c) classic geometric decreasing sum O( f(n) g(n) ).

20. f(n) is a step function sandwiched between n and . Why is “g” not true? Caution! To guarantee conclusion of the theorem, its premise must hold. The premise insists that not only f, but also g holds. f without g is possible. The following is an example: However, with Simple Analytical Functions (SAF), i.e., functions composed of only constants, arithmetic, exponential, logarithm, and functional composition, such a “bizarre” situation will not arise, and hence, the statement requiring “g” can be omitted from the premise of the theorem.

21. Simple Analytical Function (SAF) SAF : a functioncomposed of a finite number of applications of constants, arithmetic, exponential, logarithm, & functional composition. Example: FACTS: A SAF f(n) has the following properties: a) log f(n) andf(x)/x are also SAFs. b) has only a finite # of roots. (does not oscillate) c) is asymptotically sign invariant. (past its last real root) d) is asymptotically monotonic. (part (c) on its derivative) e)In ASSUME: f(n)  g(n) f) when comparing SAFs asymptotically, “n0nn0” can be replaced by the phrase “for infinitely many n” (“for i.m. n” for short). E.g., “for all n that are positive integer powers of 2”. (“past last real root”) Our short articlesays more on this topic.The next table shows a summary of ASSUME on SAF.

23. The SAFs we encounter, usually have the following simple form: for some real constants t>0, d, e. We classify S(n) for such f(n) as: A Simple form

24. RECURRENCERELATIONS

25. The Iteration Method The iteration method (or unwinding the recurrence): T(n)= f(n) + T(n –1) = f(n) + f(n-1) + T(n –2) = f(n) + f(n-1) + f(n-2) + T(n –3) = … = f(n) + f(n-1) + f(n-2) + ··· + f(3) + f(2) + T(1) = f(n) + f(n-1) + f(n-2) + ··· + f(3) + f(2) + f(1) + T(0) = f(n) + f(n-1) + f(n-2) + ··· + f(3) + f(2) + f(1). f(n) is the driving function of the recurrence Example: f(n) = Q(n)  T(n) = Q(n2). f(n) = Q(2n)  T(n) = Q(2n).

26. The Iteration Method T(n)= n + 2T(n/2) = n + 2 [n/2 + 2T(n/22)] = 2n + 22 T(n/22) = 2n + 22 [n/22 + 2 T(n/23)] = 3n + 23 T(n/23) = 3n + 23 [n/23 + 2T(n/24)] = 4n + 24 T(n/24) = … = kn + 2k T(n/2k) take k = 1+ log n, T(n/2k) = 0 = n(1 + log n)= Q(n log n)

27. The Iteration Method T(n)= n2 + 2T(n/2) = n2 + 2 [(n/2)2 + 2T(n/22)] = n2(1+ ½) + 22 T(n/22) = n2(1+ ½) + 22 [(n/22)2 + 2 T(n/23)] = n2(1+ ½ + ½2) + 23 T(n/23) = n2(1+ ½ + ½2) + 23 [(n/23)2 + 2T(n/23)] = n2(1+ ½ + ½2 + ½3) + 24 T(n/24) = … = n2 (1+ ½ + ½2+ ½3 + ··· + ½k ) + 2k+1 T(n/2k+1) take k = log n, T(n/2k+1) = 0 = n2 (1+ ½ + ½3 + ½5 + ··· + ½k ) = n2· Q(1) geometric decreasing = Q(n2 )

28. f(n) f(n) T(n/2) T(n/2) f(n/2) f(n/2) 2f(n/2) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) f(n/4) f(n/4) f(n/4) f(n/4) 4f(n/4) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) 8f(n/8) f(n/8) f(n/8) f(n/8) f(n/8) f(n/8) f(n/8) f(n/8) f(n/8) T(n/16) T(n/16) The Recursion Tree Method T(n)

29. n n T(n/4) T(n/2)  3n/4 n/4 n/2 log4 n log2 n T(n/16) T(n/8) T(n/8) T(n/4) T(n/4)  (3/4)2n n/16 n/8 n/8 n/4 T(n/64) T(n/32) T(n/32) T(n/16) T(n/32) T(n/16) T(n/16) T(n/8)  (3/4)3n n/64 n/32 n/32 n/16 n/32 n/16 n/16 n/8 T(n/256) T(n/16) The Recursion Tree Method CLAIM: T(n) = Q(n). Lower bound: T(n)  W(n) obvious T(n) Upper bound

30. The Recursion Tree Method This is a special case of the following recurrence:T(n) = T(an) + T(bn) + Q(n)where 0 a < 1 and 0 b < 1 are real constant parameters. FACT:(1)a + b < 1  T(n) = Q(n) [linear](2) a + b = 1  T(n) = Q(n log n) (3) a + b > 1  T(n) = Q(nd) [super-linear poly]where d > 1 is the unique constant that satisfies the equation ad + bd = 1. [See the guess-&-verify method later& Exercise 9.]

31. Divide-&-Conquer MergeSort is the prototypical divide-&-conquer algorithm. Here is a high level description: AlgorithmMergeSort( S ) § John von Neumann [1945] Pre-Condition: input S is a sequence of numbers Post-Condition: output is a sorted permutation of the input sequence Base: if |S|  1then returnS Divide:Divide S into its left and right halves S=L, R, |L||R| ½|S| Conquer:L’  MergeSort( L ); R’  MergeSort( R ) Combine:S’  Merge( L’, R’ ) Output:returnS’ end

32. MergeSort Example INPUT 23 15 41 82 37 92 66 31 34 73 25 19 33 15 16 58 15 23 41 82 37 92 31 66 34 73 19 25 15 33 16 58 15 23 41 82 31 37 66 92 19 25 34 73 15 16 33 58 15 23 31 37 41 66 82 92 15 16 19 25 33 34 58 73 15 15 16 19 23 25 31 33 34 37 41 58 66 73 82 92 OUTPUT

33. MERGE Loop Invariant: n merged so far A s f k s-1 i m-1 m j f+1   B right half left half Merge time = Q(n), where n = f – s +1. AlgorithmMergeSort( A[1..n] ) B[0..n+1]  auxiliary array for merging MS(1,n) end procedureMS( s, f ) § sort A[s..f] Base:if s  fthen return Divide:m (s+f)/2 Conquer:MS( s, m –1 ) ; MS( m, f ) Combine:Merge( s, m, f ) end procedureMerge( s, m, f ) B[s-1 .. m-2]  A[s,m-1] B[m .. f]  A[m,f] B[m-1]B[f+1] § barrier sentinel i  s-1 ; j  m for k  s .. f do if B[i]  B[j]then A[k]  B[i]; i++ else A[k]  B[j]; j++ end

34. AlgorithmMergeSort( A[1..n] ) B[0..n+1]  auxiliary array for merging MS(1,n) end MergeSort Recurrence: T(n) = Q(1) n  1 T(n) =T( n/2 ) + T( n/2 ) + Q(n) n>1 procedureMS( s, f ) § sort A[s..f] Base:if s  fthen return Divide:m (s+f)/2 Conquer:MS( s, m –1 ) ; MS( m, f ) Combine:Merge( s, m, f ) end procedureMerge( s, m, f ) B[s-1 .. m-2]  A[s,m-1] B[m .. f]  A[m,f] B[m-1]B[f+1] § barrier sentinel i  s-1 ; j  m for k  s .. f do if B[i]  B[j]then A[k]  B[i]; i++ else A[k]  B[j]; j++ end Simplified Recurrence: T(n) = 2 T(n/2) + Q(n) for n > 1 T(n) = Q(1) for n  1 Solution:T(n) = Q( n log n).

35. Divide-&-Conquer Recurrence a = # of recursive calls (a > 0) n/b = size of each sub-instance called recursively (b > 1) f(n) = cost of divide + combine steps (every thing except the recursive calls). In MergeSort we have: a = 2 recursive calls of size n/b = n/2 each. Cost of divide = Q(1) Cost of Combine, i.e., merge, is Q(n). So, f(n) = Q(n).

36. f(n) f(n) a af(n/b) f(n/b) f(n/b) f(n/b) a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) f(n/b2) k = logbn f(n/bk-1) ak-1f(n/bk-1) f(n/bk-1) h(n) = Q(n logba) Q(1) Q(1) Q(1) Q(1) a k Recursion Tree: a logb n = n logb a(compare their logarithms) h(n) = ak ·Q(1) = Q(ak) =Q(alogbn) =Q(alogbn) = Q(nlogba) k = logbn = Q( logn).

37. S(n) = S f(n) T(n) = S(n) + h(n) = Q(max{S(n), h(n)}) h(n) = Q( n logba) af(n/b) a2f(n/b2) ak-1f(n/bk-1) h(n) = Q(n logba) a logb n = n logb a(compare their logarithms) h(n) = ak ·Q(1) = Q(ak) =Q(alogbn) =Q(alogbn) = Q(nlogba) k = logbn = Q( logn).

38. Special Solution: contribution of the internal nodes of the recursion tree. Homogeneous Solution: contribution of the leaves of the recursion tree. h(n) = Q(n logba) T(n) = S(n) + h(n) T(n) = h(n)( Q(n) + 1 ) f(n) is a SAF r(n) is a SAF

39. T(n) = h(n)( Q(n) + 1 ) f(n) is a SAFr(n) is a SAF Let n = bk for integer k.

40. h(n) = Q( n logba ) THE MASTER THEOREM Suppose f(n) is a SAF, and e > 0 & e are constant reals.

41. h(n) = Q( n logba )

42. CSE 3101 Sums & Recurrences Fact Sheet

43. h(n) = Q( n logba ) Example 1: Assume a = 2, b = 4, f(n) = Q(nd log2 n). Express T(n) based on the constant parameter d. Solution:logb a = log4 2 = ½  h(n) = Q() d > ½ T(n) = Q(f(n)) = Q() d = ½ T(n) = Q(f(n) log n) = Q() d < ½ T(n) = Q(h(n)) = Q()

44. h(n) = Q( n logba ) Example 2: Assume b = 3,f(n) = Q(n4 / log n). Express T(n) based on the constant parameter a. Solution:h(n) = Q(nlog3a) , bd = 34 = 81 a < 81 T(n) = Q(f(n)) = Q(n4 / log n) a = 81 T(n) = Q(h(n) log log n) = Q(n4 log log n) a > 81 T(n) = Q(h(n)) = Q(nlog3a)

45. Guess-&-Verify BASIC INGRIDIANTS: • Guess a solution to the recurrence. • Verify it by mathematical induction. The induction variable must be a natural number. E.g., height of the recursion tree (max recursion depth), or size of the recursion tree (# times you apply the recurrence). n may also be a candidate if it “stays” integer. • If you spot a place where the verification does not go through, examine it to help revise your guess, and try again. • One key guiding principle, taken with a grain of salt, is to first make sure the leading term (with its constant coefficient) matches on the LHS and RHS of the recurrence. That will show you to guess higher or lower the next time! After figuring out the leading term, you can apply the same principle to find the lower order terms.

46. we need c  0 Guess-&-Verify: Example Recurrence: T(n) = 2T( n/2 ) + n2 , T(0) = 0 T(n)  n2 W(n2) is obvious. Let’s see if O(n2) is an upper-bound: Guess: T(n)  an2 + bn + c(for some constants a>0, b, c to be determined) Basis (n = 0): T(0) = 0 a02 + b0 + c Ind. Step (n 1): T(n) = 2T( n/2 ) +n2  2 [ a n/22 + b n/2 + c ] + n2 by induction hypothesis if n is even= 2 [ a(n/2)2 + b(n/2)+c ] + n2 not “” for odd n, if b<0 = (1+ a/2)n2 + bn + 2c  an2 + bn + c our guessed upper-bound We need: for all even n 1:an2 + bn + c  (1+ a/2)n2 + bn + 2c, i.e.,(a/2 – 1)n2 – c  0.

47. we need c  0 a = 2 b = -1 c = 0 works for all n  0. n2  T(n)  2n2 - n Guess-&-Verify: Example Recurrence: T(n) = 2T( n/2 ) + n2 , T(0) = 0 T(n)  n2 W(n2) is obvious. Let’s see if O(n2) is an upper-bound: Guess: T(n)  an2 + bn + c(for some constants a>0, b, c to be determined) Basis (n = 0): T(0) = 0 a02 + b0 + c Ind. Step (n 1): T(n) = 2T( n/2 ) +n2  2 [ a n/22 + b n/2 + c ] + n2 by induction hypothesis if n is odd = 2 [ a((n-1)/2)2 + b((n-1)/2)+c ] + n2 = (1+ a/2)n2 + (b-a)n + (2c-b+a/2)  an2 + bn + c our guessed upper-bound We need: for all odd n 1:an2 + bn + c  (1+ a/2)n2 + (b-a)n + (2c-b+a/2), i.e.,(a/2 – 1)n2 + an + (b – c – a/2)  0. We need: for all even n 1:an2 + bn + c  (1+ a/2)n2 + bn + 2c, i.e.,(a/2 – 1)n2 – c  0.

48. Guess-&-Verify: Example Recurrence: T(n) = 2T( n/2 ) + n2 , T(0) = 0 Exercise 8: Using this same method, verify the following tighter LOWER BOUND: n1: 2n2 – n – 2n log n  T(n)  2n2 – n.  T(n) = 2n2 – n – O(n log n). a = 2 b = -1 c = 0 works for all n  0. n2  T(n)  2n2 - n For more examples of guess-&-verify see Lecture Note 2 & Sample Solutions ….

49. This is the QuickSort expected time recurrence. (We shall show that later). Note this recurrence is discrete version of the continuous integral equation Full History Recurrence Such recurrences often arise in average-case analysis. Values of T(n) for some small n: T(0) = f(0) T(1) = T(0) + f(1) = f(0) + f(1) T(2) = T(0) + T(1) + f(2) = 2f(0) + f(1) + f(2) T(3) = T(0) + T(1) + T(2) + f(3) = 4f(0) + 2f(1) + f(2)+ f(3) T(4) = T(0) + T(1) + T(2) + T(3) + f(4) = 8f(0) + 4f(1) + 2f(2) + f(3) + f(4)

50. Full History Recurrence A Solution Method: Now there is only one appearance of T(.) on the RHS. Continue with conventional methods.