::ICS 804:: Theory of Computation - Ibrahim Otieno - iotieno@uonbi.ac.ke +254-0722-429297

1. ::ICS 804:: Theory of Computation - Ibrahim Otieno - iotieno@uonbi.ac.ke +254-0722-429297 SCI/ICT Building Rm. G15

2. Course Outline • Mathematical Preliminaries • Turing Machines • Recursion Theory • Markov Algorithms • Register Machines • Regular Languages and finite-state automata • Aspects of Computability

3. Last week: Regular Languages and FSA • Regular Expressions and regular languages • Deterministic FSA • Non-deterministic FSA • Finite-state automata with epsilon moves • Generative grammars • Context-free, Context-sensitive languages • Chomsky Hierarchy

4. Course Outline • Mathematical Preliminaries • Turing Machines • Additional Varieties of Turing Machines • Recursion Theory • Markov Algorithms • Register Machines • Regular Languages and finite-state automata • Aspects of Computability

5. Aspects of Computability • Church-Turing Thesis • Rice’s Theorem • Wrap-Up

6. The Church-Turing Thesis

7. Church-Turing • Effectively calculable function: cultural notion; part of mathematician’s informal sense of what mathematics is about • Turing-computable function: not cultural or informal at all • Church-Turing Thesis tries to combine these two notions (focus on number-theoretic functions)

8. Church-Turing • Turing (1936): a number-theoretic function f is effectively calculable if and only if f is Turing-computable • Not possible to formally prove it. Why not? • Try to justifyit • Two parts: • f is effectively calculable so f is Turing-computable • f is Turing-computable so f is effectively calculable • Obvious: there exists some algorithm, so it is calculable

9. Church-Turing • f is effectively calculable so f is Turing-computable: not trivial • Does there exist a function that is effectively calculable but not Turing-computable? • Many (very different) models of computation, all related (cf. equivalence results) • Turing computable functions, Markov-computable functions, … • Partial Recursive Functions!

10. Feasibility (recap) • Cobham–Edmonds thesis regarding computational feasibility: the problem of determining whether a given string is a member of a given language L is feasible if and only if L is in P Complexity class P: class of all languages accepted in polynomial time by some (single-tape) Turing Machine

11. Problems with the Thesis • Computable by what? Focus on single-tape Turing machine • Which model of computation is the favored one? • Does it make a difference?

12. Problems with the Thesis e.g. L = {w*|na(w) = nb(w)} • Register machine: O(n) steps • Single Tape Turing Machine: O(n2) steps • Multi-Tape Turing Machine: O(n) steps • Markov Machine: O(n2) steps

13. Polynomial Relatedness • Suppose that multitape MTM accepts language L in O(n) steps. Then there exists MRM that accepts L in time O([timeTM(n)]2) under the uniform cost assumption. • Suppose that MRM accepts language L in O(n) steps under the assumption of uniform cost. Then there exists multitape MTM that accepts L in time O([timeTM(n)]3).

14. Complexity (recap: MA) • Given Turing machine M accepting L, there exists a Markov algorithm AS that accepts L in O(timeM(n)) steps • Given Markov algorithm AS accepting L, there exists a Turing machine M that accepts L in O([timeAS(n)]4) steps

15. Time and Space (recap: TM) • Corollary Suppose that language L is accepted by k-tape Turing machine M. Then L is accepted by a single-tape Turing machine M1 such that timeM1(n) is O([timeM(n)]2) • Corollary Suppose that language L is accepted by k-tape Turing machine M. Then L is accepted by a single-tape Turing machine M´ such that spaceM´(n) is O(spaceM(n)).

16. Results Regarding Polynomial Relatedness of Models Markov Algorithm Model O(timeM(n)) O([timeAS(n)]4) Multitape Turing Machine model O([timeTM(n)]3) O(timeM(n)) O([timeM(n)]2) Register Machine Model Single-tape Turing Machine model O([timeTM(n)]2)

17. Reformulation • In invoking the Cobham-Edmonds thesis, it seems to make no difference which model we take Reformulation: A language-acceptance problem is feasible provided that the language in question is accepted in polynomially bounded time

18. More generally Invariance principle for models of sequential Computation Let Model be some machine model of sequential computation. Then Model will be said to be a reasonable model of sequential computation if it is polynomially related to the multitape Turing machine model

19. Church-Turing • Over 7 decades, many different models of computation, but all can be related to the Turing-machine model • All seem to encompass the same class of functions: partial recursive functions • Hypothesis: • this must be a “natural” class • Mathematical properties of its members make it the natural outcome of diverse approaches • This class must be the class of effectively calculable functions • If one of the other models could compute some function that is not Turing-computable, then the class would not seem to be so properly defined

20. Church-Turing • So: no formal proof, but vast amount of empirical evidence Thesis: If function f is effectively calculable, then f is Turing computable. Equivalently, if a function f is not Turing computable, then f is not effectively calculable • But limits of human cognition?

21. Rice’s Theorem

22. Rice’s Theorem • Preliminaries: • Every natural number is the encoding or gödel number of a unique single-tape deterministic Turing machine • Every Turing machine with input alphabet ={1} computes a unary partial recursive function • The class of Turing machines with input alphabet ={1} can be effectively enumerated

23. Recursive Set (recap) A set S of natural numbers is recursive if its characteristic function S is (partial) recursive with S(n) = 1 if n  S 0 if n  S

24. Rice’s Theorem Theorem: Let  be a set of unary partial recursive functions that is nontrivial (i.e. not empty or the entire class). Then let  be the set of all gödel numbers of the members of . Then  is not a recursive set of natural numbers

25. Rice’s Theorem Important ramifications for pragmatic computer science: e.g. teacher trying to grade C programs for some assignment • Approach 1: check input/output • Approach 2: grading program that runs on source code to check whether it effectively computes some function • Following Rice’s Theorem (and Church-Turing) there cannot be such a grading program

26. Wrap-up

27. Paradigms • Three computational paradigms: • Function computation • Language recognition • Transduction paradigm • Various ways of modeling these paradigms: Turing machines, Markov algorithms, register machines

28. Turing • Starting point: single-tape Turing machines • Variations: one-way-infinite tape, halting state, multi-tape • Inconsequential from theoretical point of view • Variants: extensionally equivalent

29. Markov Algorithms • Labeled and unlabeled • Like Turing machines, universal model of computation • Universal: • Not: can do everything • But: any function that is computable in any sense is computed by some Markov algorithm • Church-Turing: functions that are not Markov-computable are not computable in any intuitive sense either

30. Computability-in-principle • Computation without limitations in terms of time and resources • Some functions are computable in principle, but may take ages to finish • e.g. Busy Beaver is not Turing-computable

31. Busy Beaver • The Turing Machine in TMn that writes the maximum number of 1s for a number of states • Very hard to find for anything but the smallest of n’s • For any n: large number of possible Turing Machines • Some Turing Machines will loop (so (n) = 0) • Hard to distinguish between infinitely looping machine and one that is just writing a lot of 1s

32. Busy Beaver • Calculated by brute force

33. Busy Beaver • Very theoretic, toy-like argument to non-computability • But: impact on pragmatic computer science as well • Rice’s theorem: various decision problems are not solvable by any Turing machine and thus not solvable at all • Various algorithms for proving program correctness do not exist: computer scientists with no background in theory can waste a lot of time trying to develop such an algorithm

34. Theory of Computation • Origins in 1920s/1930s by mathematical logicians and philosophers before the advent of the modern digital computer!!! • Theoretic computer science has been advancing independently from pragmatic computer science • But: theoretic work has influenced pragmatic developments and vice versa e.g. Universal Turing Machine as a precursor to modern-day Computability theory influences design of programming languages and compilers Register machines: reflect modern machine architecture

35. Theory of Computability • A side-branch of mathematics • But: mathematicians cannot axiomatically characterize mathematical truth • Church-Turing, Cobham-Edmonds, … try to define computability and seem to have largely succeeded • Applications in pragmatic computer science have validated these insights • But: theory of computation is a science in its own right (e.g. theoretical physics, theoretical linguistics, cognitive science, …) • What is computable? • What is feasible? • Basic philosophical questions!

36. Feasible Computation • More interesting from a practical point of view • Complexity theory can aid during program design • Cobham-Edmonds Thesis: feasibility is modeled by a deterministic, single-tape Turing machine whose running time is bounded above by some polynomial in the size of the input • Complexity class P (and PNP): can be computed in polynomial time • Complexity class NP: requires exponential time • NP-complete problems not in the class of P are deemed to be computationally unfeasible

37. Feasible Computation • So when working on a problem: if you find the solution to not be solvable in polynomial time, direct your effort to a subset of the problem • E.g. traveling salesman problem for Weighted undirected graphs  refocus the problem with some heuristics, e.g. nearest-neighbor traveling salesman problem • So complexity theory has had a huge impact outside of theoretical computing

38. Feasible Computation • P-completeness: although computation is feasible, the problem is unlikely to be solvable using a parallel computing approach • in other words: a parallel algorithm will not produce any improvement in worst-case execution time over the polynomially bounded non-parallel algorithm that already exists • So knowing this: if a problem is in complexity class P and is polynomially bounded in time, don’t waste time trying to parallelize the solution

39. Part II: The Chomsky Hierarchy Accepted by nondeterministic push-down stack automaton G: Context-Free Grammars Accepted by linear-bounded automaton G: context-sensitive grammars Accepted by deterministic FSA G: Right-linear grammars Recursively enumerable languages Context-free languages Context- sensitive languages Regular languages Accepted by deterministic 1tape Turing Machine

40. Open questions • P = NP ? • Can deterministic linear-bounded automaton accept the same language as a nondeterministic one? • Nondeterminism, although proven to be inconsequential for Turing Machines and finite-state automata, is still not fully understood for models of computation