Chapter 3

1. Chapter 3 • 3.1 Algorithms • 3.2 The Growth of Functions • 3.3 Complexity of Algorithms • 3.4 The Integers and Division • 3.5 Primes and Greatest Common Divisors • 3.6 Integers and Algorithms • 3.7 Applications of Number Theory • 3.8 Matrices

2. Chapter 3 • 3.1 Algorithms • Searching Algorithms • Greedy Algorithms • The Halting Problem

3. Algorithm • Definition 1: An algorithm is a finite set of precise instructions for performing a computation or for solving a problem. • Example 1: Describe an algorithm for finding the maximum (largest) value in a finite sequence of integers.

4. We perform the following steps • Set the temporary maximum equal to the first integer in the sequence. (the temporary maximum will be the largest integer examined at any stage of the procedure.) • Compare the next integer in the sequence to the temporary maximum, and if it is larger than the temporary maximum, set the temporary maximum equal to this integer. • Repeat the previous step if there are more integers in the sequence. • Stop when there are no integers left in the sequence. The temporary maximum at this point is the largest integer in the sequence.

5. Pseudocode • Pseudocode provides an intermediate step between an English language description of an algorithm and an implementation of this algorithm in a programming language. • Algorithm 1: Finding the maximum element in a finite sequence. procedure max(a1, a2, . . . ,an: integers) max := a1 fori: =2 to n if max < aithen max := ai {max is the largest element}

6. Property of Algorithm • Input. • Output. • Definiteness. The steps of an algorithm must be defined precisely. • Correctness. • Finiteness. • Effectiveness. • Generality. The procedure should be applicable for all problems of the desired form, not just for a particular set of input values.

7. Searching Algorithms Search Problem: Locating an element in an (ordered) list. • Linear search • Binary search (ordered list)

8. The linear search • Algorithm 2 : the linear search algorithm procedure linear search (x: integer, a1, a2, …,an: distinct integers) i :=1; while ( i ≤n and x ≠ ai) i:= i + 1 Ifi≤ nthenlocation := i Else location := 0 {location is the subscript of the term that equals x , or is 0 if x is not found}

9. The binary search • Algorithm 3: the binary search algorithm Procedure binary search (x: integer, a1, a2, …,an: increasing integers) i :=1 { i is left endpoint of search interval} j :=n { j is right endpoint of search interval} While i < j begin m := (i+j)/2 if x > am then i := m+1 else j := m end If x = ai then location := i else location :=0 {location is the subscript of the term equal to x, or 0 if x is not found} Example 3: to search for 19 in the list 1 2 3 5 6 7 8 10 12 13 15 16 18 19 20 22

10. Sorting • Sort: • Sorting is putting elements into a list in which the elements are in increasing order. • E.g. • 7,2,1,4,5,9 -> 1,2,4,5,7,9 • d,h,c,a,f -> a,c,d,f,h. • Bubble sort • Insertion sort

11. Bubble Sort • ALGORITHM 4: The Bubble Sort procedurebubble sort (a1, a2, …,an: real numbers with n ≥2) fori := 1 to n-1 for j := 1 to n- i if aj > aj+1 then interchange aj and aj+1 {a1, a2, …,anis in increasing order} • Example 4: Use the sort to put 3, 2, 4, 1, 5 into increasing order.

12. Bubble Sort

13. Insertion Sort • Algorithm 5: The Insertion Sort procedure insertion sort (a1, a2, …,an: real numbers with n ≥2) for j := 2 to n begin i:= 1 while aj> ai i := i + 1 m := aj for k :=0 to j-i-1 aj-k := a j-k-1 ai := m end {a1, a2, …,anare sorted} Example 5: Use the insertion sort to put the elements of the list 3, 2, 4, 1, 5 into increasing order.

14. Greedy Algorithm • Optimization Problem: find the best solution. • Algorithms that make what seems to be the best choice at each step are called greedy algorithms.

15. Example 6: Consider the problem of making n cents change with quarters, dimes, nickels, and pennies, and using the least total number of coins. • Algorithm 6: Greedy Change-Marking Algorithm procedure change (c1, c2, …, cr: values of denominations of coins, where c1> c2> … > cr; n: a positive integer) fori := 1 to r while n ≥ ci begin add a coin with value ci to the change n := n – ci end

16. The Halting Problem • There is a problem that cannot be solved using any procedure. • That is, there are unsolvable problems. • Halting Problem FIGURE 2 Showing that the Halting Problem is Unsolvable.

17. Chapter 3 • 3.1 Algorithms • 3.2 The Growth of Functions • 3.3 Complexity of Algorithms • 3.4 The Integers and Division • 3.5 Primes and Greatest Common Divisors • 3.6 Integers and Algorithms • 3.7 Applications of Number Theory • 3.8 Matrices

18. Chapter 3 • 3.2 The Growth of Functions • Big-O Notation • Some Important Big-O Results • The Growth of Combinations of Functions • Big-Omega and Big-Theta Nation

19. The Growth of Functions We quantify the concept that g grows at least as fast as f. What really matters in comparing the complexity of algorithms? • We only care about the behavior for large problems. • Even bad algorithms can be used to solve small problems. • Ignore implementation details such as loop counter incrementation, etc. we can straight-line any loop.

20. Big-O Notation • Definition 1: let f and g functions from the set of integers or the set of real numbers to the set of real number. We say that f(x) is O(g(x)) if there are constants C and k such that |f(x)| ≤ C |g(x)| whenever x > k. • This is read as “ f(x) is big-oh of g(x) ”. • The constants C and k in the definition of big-O notation are called witnesses to the relationship f(x) is O(g(x)). • Note: • Choose k • Choose C ; it may depend on your choice of k • Once you choose k and C, you must prove the truth of the implication (often by induction). • Example 1: show that f(x)= x2+ 2x + 1 is O(x2)

21. Big-O Notation FIGURE 1 The Function x2 + 2x + 1 is O(x2).

22. Big-O Notation FIGURE 2 The Function f(x) is O(g(x)).

23. Big-O Notation • Example 2: show that 7x2 is O( x3 ). • Example 4:Is it also true that x3 is O(7x2)? • Example 3: show that n2 is not O(n).

24. Little-O Notation • An alternative for those with a calculus background: • Definition: if then f is o(g), called little-o of g.

25. Theorem: if f is o(g) then f is O(g). • Proof: by definition of limit as n goes to infinity, f(n)/g(n) gets arbitrarily small. That is for any ε >0 , there must be n integer N such that when n > N, | f(n)/g(n) | < ε. Hence, choose C = εand k= N. Q.E.D. It is usually easier to prove f is o(g) • Using the theory of limits • Using L’Hospital’s rule • Using the properties of logarithms etc

26. Example : 3n + 5 is O(n2). • Proof: it’s easy to show using the theory of limits. Hence, 3n+5 is o(n2) and so it is O(n2). Q.E.D.

27. Some Important Big-O Results • Theorem 1: let where a0, a1, . . .,an-1, an are real numbers then f(x) is O(xn) . • Example 5: how can big-O notation be used to estimate the sum of the first n positive integers?

28. Some Important Big-O Results • Example 6: give big-O estimates for the factorial function and the logarithm of the factorial function, where the factorial function f(n) =n! is defined by n! = 1* 2 * 3 * . . .*n Whenever n is a positive integer, and 0!=1.

29. Some Important Big-O Results • Example 7: In Section 4.1 ,we will show that n <2n whenever n is a positive integer. Show that this inequality implies that n is O(2n) , and use this inequality to show that log n is O(n).

30. The Growth of Combinations of Functions • 1 • logn • n • n logn • n2 • 2n • n! FIGURE 3 A Display of the Growth of Functions Commonly Used in Big-O Estimates.

31. Important Complexity Classes Where j > 2 and c> 1. • Example :Find the complexity class of the function • Solution: this means to simplify the expression. Throw out stuff which you know doesn’t grow as fast. We are using the property that if f is O(g) then f + g is O(g).

32. Important Complexity Classes if a flop takes a nanosecond, how big can a problem be solved (the value of n ) in a minute? a day? a year? For the complexity class O(n n! nn)

33. Important Complexity Classes a minute= 60*109= 6*1010 flops a day= 24*60*60= 8.65*1013 flops a year= 365*24*60*60*109= 3.1536*1016 flops We want to find the maximal integer so that n*n!*nn < 6*1010 n*n!*nn < 8.65*1013 n*n!*nn < 3.1536*1016

34. Important Complexity Classes Maple Program: for k from 1 to 10 do (k,k*factorial(k)*kk)end do; 1, 1 2, 16 3, 486 4, 24576 5, 187500 6, 201553920 7, 29054597040 8, 5411658792960 9, 1265284323434880 10, 362880000000000000 So, n=7,8,9 for a minute, a day, and a year.

35. The Growth of Combinations of Functions • Theorem 2: suppose that f1(x) is O(g1(x)) and f2(x) is O(g2(x)).Then (f1 + f2)(x) is O(max( |g1(x)| , |g2(x)| )). • Corollary 1: suppose that f1(x) and f2(x) are both O(g(x)). Then (f1 + f2)(x) is O(g(x)).

36. Theorem: If f1 is O(g1) and f2is O(g2) then • f1 f2 is O(g1g2) • f1+f2 is O(max {g1 ,g2})

37. The Growth of Combinations of Functions • Theorem 3 :suppose that f1(x) is O(g1(x)) and f2(x) is O(g2(x)). Then (f1f2)(x) is O(g1(x) g2(x)). • Example 8: give a big-O estimate for f(n)=3n log(n!) + (n2 +3) log n where n is a positive integer. • Example 9: give a big-O estimate for f(x)=(x+1)log(x2+1) + 3x2

38. Properties of Big-O • f is O(g) iff • If f is O(g) and g is O(f) then • The set O(g) is closed under addition: if f is O(g) and h is O(g) then f+h is O(g) • The set O(g) is closed under multiplication by a scalar a (real number):if f is O(g) then af is O(g) That is ,O(g) is a vector space. (The proof is in the book.) Also, as you would expect, • If f is O(g) and g is O(h), then f is O(h) . In particular

39. Note : we often want to compare algorithms in the same complexity class • Example: Suppose Algorithm 1 has complexity n2 – n +1 Algorithm 2 has complexity n2/2 + 3n + 2 Then both are O(n2) but Algorithm 2 has a smaller leading coefficient and will be faster for large problems. Hence we write Algorithm 1 has complexity n2 +O(n) Algorithm 2 has complexity n2/2 + O(n)

40. Big-Omega and Big-Theta Nation • Definition 2: Let f and g be functions from the set of integers or the set of real numbers to the set of real numbers. • We say that f(x) is Ω(g(x)) if there are positive constants C and k such that |f(x)|≥ C|g(x)| Whenever x > k. ( this is read as “f(x) is big-Omega of g(x)” .) • Example 10 :The function f(x) =8x3+ 5x2 +7 is Ω(g(x)) , where g(x) is the function g(x) =x3. • This is easy to see because f(x) =8x3+ 5x2 +7 ≥ x3 for all positive real numbers x. this is equivalent to saying that g(x) = x3 is O(8x3+ 5x2 +7 ) ,which can be established directly by turning the inequality around.

41. Definition 3: Let f and g be functions from the set of integers or the set of real numbers to the set of real numbers. • We say that f(x) is Θ(g(x)) if f(x) is O(g(x)) and f(x) is Ω(g(x)). • When f(x) is Θ(g(x)) , we say that” f is big-Theta of g(x)” and we also say that f(x) is of order g(x). • Example 11: we showed (in example 5) that the sum of the first n positive integers is O(n2). Is this sum of order n2? • Example 12: show that 3x2 + 8x(logx) isΘ(x2).

42. Theorem 4: let , where a0, a1, . . .,an-1, an are real numbers with an≠0 . Then f(x) is of order xn. • Example 13: the ploynomials 3x8+10x7+221x2+1444 x19-18x4-10112 -x99+40001x98+100003x are of orders x8, x19 and x99 ,respectively.

43. Chapter 3 • 3.3 Complexity of Algorithms • Time Complexity • Understanding the complexity of Algorithms

44. Complexity of Algorithm • Computational Complexity (of the Algorithm) • Time Complexity: Analysis of the time required. • Space Complexity: Analysis of the memory required.

45. Time Complexity • Example 1: Describe the time complexity of Algorithm 1 of section 3.1 for finding the maximum element in a set (in terms of number of comparisons). • Algorithm 1: Finding the maximum element in a finite sequence. procedure max(a1, a2, . . . ,an: integers) max := a1 fori: =2 to n if max < aithen max := ai {max is the largest element}

46. Example 2: Describe the time complexity of the linear search algorithm. • Algorithm 2 : the linear search algorithm procedure linear search (x: integer, a1, a2, …,an: distinct integers) i :=1; while ( i ≤n and x ≠ ai) i:= i + 1 Ifi≤ nthenlocation := i Else location := 0 {location is the subscript of the term that equals x , or is 0 if x is not found}

47. Example 3: Describe the time complexity of the binary search algorithm in terms of the number of comparisons used . (and ignoring the time required to compute m= in each iteration of the loop in the algorithm) • Algorithm 3: the binary search algorithm Procedure binary search (x: integer, a1, a2, …,an: increasing integers) i :=1 { i is left endpoint of search interval} j :=n { j is right endpoint of search interval} While i < j begin m := if x > am then i := m+1 else j := m end If x = ai then location := I else location :=0 {location is the subscript of the term equal to x, or 0 if x is not found}

48. Example 4: Describe the average-case performance of the linear search algorithm, assuming that the element x is in the list. • Example 5: What is the worst-case complexity of the bubble sort in terms of the number of comparisons made? • ALGORITHM 4: The Bubble Sort procedurebubble sort (a1, a2, …,an: real numbers with n ≥2) fori := 1 to n-1 for j := 1 to n- i if aj > aj+1 then interchange aj and aj+1 {a1, a2, …,an is in increasing order}

49. Example 6: What is the worst-case complexity of the insertion sort in terms of the number of comparisons made? • Algorithm 5: The Insertion Sort procedure insertion sort (a1, a2, …,an: real numbers with n ≥2) for j := 2 to n begin i := 1 while aj> ai i := i + 1 m := aj for k :=0 to j-i-1 aj-k := a j-k-1 ai := m end {a1, a2, …,an are sorted}

50. Understanding the complexity of Algorithms