Algorithmic Problems

1. Algorithmic Problems Zeph Grunschlag

2. Agenda • Viewing algorithms as languages • Algorithmic Decision Problems • ATM, ACFG , AREX, ETM, … etc.

3. Knowing the Unknowable Arrived at important juncture in course. Now we will study one of the fundamental questions in Computer Science: What is the ultimate power of computational machinery? Of equal importance in human quest to understand self: What are the impenetrable limits of human knowledge?

4. Knowing the Unknowable Accepting the Church-Turing thesis implies that these are the same question! Church-Turing thesis is usually recognized as applying to human computations, and therefore human thought processes. Amazing Fact: It is possible to give precise examples of problems which are beyond the reach of mechanical computation as well as human intellect. We in fact “know the unknowable.”

5. Decision Problems A decision problem is a set of yes/no questions, each called an instance. For the decision problem to be interesting, the set of questions should be infinite. EG: “Does z equal the sum of x and y ?” is a good example. A computer program that can give the correct solution for any possible x,y,z intuitively must have figured out how to add numbers. EG: “Does 8 equal the sum of 3 and 5?” is a bad example. Could easily write program for this instance without knowing how to add!

6. Decision Problems Q: How about the problem “Does God Exist?” Can a computer solve this problem?

7. Decision Problems A: In the way defined above, the answer is yes! We give two computer programs, both of which accept the input “Does God exist?” One of these programs is correct! Therefore, that computer program has solved the problem of deciding if God exists! Q: Any misgivings? A(String question){ return “No!” } B(String question){ return “Yes!” }

8. Decision Problems A: Many possible misgivings. Here’s one: “Solution” to existence of God surely not what is meant by solving the problem. We expected the solution to be arrived at by careful consideration of physical evidence, or by theological arguments, or some other intricate process. CS viewpoint: Any algorithm which works only on a finite set of inputs is considered meaningless, as could always program a look-up table which gave correct solutions without using any insight.

9. Decision Problems So to solve the deity existence problem would have to give a more general solution. EG: Given: A complete description of a physical universe and a complete description of a deity. Decide: Does the deity necessarily exist for the given universe? Now there are infinitely many possible inputs, so cannot just use a look-up table and this is a meaningful problem, though hopeless…

10. Decision Problems as Languages To study what it means for a decision problem to be solvable, need a method of standardizing such problems and making them easily understood by computers. Can’t just come over to a computer and ask “Does 2 + 2 = 4?” and even if could, dealing with voice recognition, natural language processing, etc. takes us far afield to topics irrelevant to answering “is the problem solvable?” Solution: represent each instance by a string: Thus instance is the string “Does 2 + 2 = 4?” or more simply “2+2=4”. Q: Define solvability language theoretically?

11. Decision Problems as Languages A: A decision problem P is solvable if its encoding by strings admits a computer program which inputs such encodings, always halts, and always outputs the correct answer. Furthermore, the set of encodings should be computable in its own right. Language theoretically: Let S be the alphabet in which the encoded instances of P exist. Let E be the set of encoded instance and let L be the set corresponding to the YES instances.1 Then P is said to be decidable (or algorithmically solvable or computable) if L is decidable.

12. Decision ProblemsSensible Encodings Some common sense is needed: The encoding scheme should be sensible but there is no way to define what sensible means. EG: “Is p a prime number?” • Encoding scheme which results in the positive instances L = {2,3,5,7,…} is sensible. • Encoding scheme which results in the positive instances L = {1,4,6,8,9,10,…} is not!

13. Decision ProblemsSensible Encodings Sensible encoding of sensible decision problem would never result in an undecidable encoding set E but condition still needed so don’t solve impossible problems unwittingly. EG: “Does a given Java program with no infinite loops print out “I’m not loopy!” on some input?” Natural encoding set E is over Unicode alphabet, and is just the set of Java programs with no infinite loops. But we’ll see that E is itself undecidable so second part becomes meaningless.

14. Decision ProblemsBasic Problems Before tackling fundamental decision problems of CS, start introspectively by considering if there are algorithms for… • …telling if a given string is accepted by a given language of a fixed type C such as DFA’s, NFA’s, PDA’s etc. Acceptance ProblemAC • …telling if a given language of type is C empty. Emptiness ProblemEC • …telling if two given languages of the same type C are equal. Equivalence ProblemEQC Each in turn, is encoded by a language.

15. Decision ProblemsEncoding Example Let’s see how ADFA may be encoded. ADFA is the problem of deciding Given: A DFA M and a string x. Decide: Does M accept x ? The language L of interest is the language of encodings of pairs (M, x ) where M accepts x. Encoding is denoted using the angled braces notation “<M, x >”. In practice, will never really work with encodings as too unwieldy. However, for once it’s worth to see how this might actually be carried out.

16. Decision ProblemsEncoding Example a Consider the DFA: Q: How can you represent this by a string? 0,1 0,1 b

17. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) 0,1 0,1 b

18. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) Q 0,1 0,1 b

19. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) S 0,1 0,1 b

20. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) d 0,1 0,1 b

21. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) q0 0,1 0,1 b

22. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) F 0,1 0,1 b

23. Decision ProblemsEncoding Example a A: DFA’s are 5-tuples, so just write down the 5 tuple, spelling out all details. Therefore <M > = ({a,b},{0,1},{(a,0,b),(a,1,b),(b,0,a),(b,1,a)},a,{a}) Encoding <M,w > amounts to just adding the comma and the input string w. Q: What’s the alphabet of encoding lang. E ? 0,1 0,1 b

24. Decision ProblemsEncoding Example A: TOO BIG! The number of letters in alphabet and number of states is unbounded. Using different characters for each state/letter would require an infinite alphabet if want to be able to encode every possible DFA! Instead, should fix some alphabet (e.g. binary or ASCII) and refer to states/letters using numbers. Thus the state a is denoted by 1, since it’s the first state, the state b by 2 and so on…

25. Decision ProblemsWhat You Really Do! After going to all this effort to define decision problems in terms of languages, Turing machines, encodings and so on, in practice, we use none of the above. The point was to argue that in principle this could be done, and remind ourselves that this ought to be achievable, when wondering whether a not purported pseudocode is really an algorithm. Sipser’s approach tries to maintain the veneer of this approach. I find this to be artificial at times so will just work directly with the natural data structure inherited by the problem.

26. ADFA For example, the easiest way to view a DFA is as a labeled graph. Encoding the DFA by a string and trying to apply some graph theoretic algorithm to the representative string is overly burdensome. You basically have to recreate the whole graph on the TM tape, which in principle can be done, but in practice is silly, especially since the starting point was a graph! Q: How do you solve ADFA ?

27. ADFA A: Just follow the path labeled by the input string from the start state. Accept if end was in F. More precisely. DFAaccept(DFA M, String x1 x2 x3 … xn ) State q = q0 //q0 defined by M’s 5-tuple for(i = 1 to n) q = d(q,xi) // d defined by M’s 5-tuple if(q  F) // F defined by M’s 5-tuple return ACCEPT else return REJECT

28. ANFA Q: How about ANFA?

29. ANFA A: Just determinize first and use solution to ADFA. NFAaccept(NFA N, String x1 x2 x3 … xn ) DFA M = determinize(N) //Subset Constr’n return DFAaccept(M, x1 x2 x3 … xn ) Q1: What’s the running time? Q2: What’s not ideal about this algorithm? Q3: What’s the fix?

30. ANFA A1: O(2|N|+n) since determinizing might involve all the subsets of the states of N so is exponential, but once have determinized, simulating the DFA is O(n). Q2: The problem is the exponential running time. Q3: We already know the fix. It’s just the game “Determinize” which keeps track of active state without explicitly constructing all the subsets. Running time of O(|N |2 ·n) since each update of the active states may involve looking at O(|N |) active states each of which may activate O(|N |) other states.

31. ANFA NFAaccept2(NFA N, String x1 x2 x3 … xn ) StateSet S = {q0 } // store S as bit string for(i = 1 to n) S = d(S,xi) // since NFA, d outputs a set if(S and F are not disjoint) return ACCEPT else return REJECT

32. EDFA Let’s now focus on emptiness problem. We can use the pumping lemma to solve the emptiness problem: Suppose we know what the pumping number p is for our DFA. Then if we found no accepted strings of length < p we could be sure that the DFA accepts no strings. Q: Why is this true?

33. EDFA A: Suppose language not empty. Since checked that all strings of length < p are rejected, smallest accepted string s must be of length  p. The string s is pumpable, so can be pumped down obtaining a shorter string s’, which contradicts the minimality of s ! This prompts the following algorithm:

34. EDFA DFAempty( DFA M ) integer p = |Q | for (all strings x over S with length < p) if( DFAaccepts(M,x) == ACCEPT ) return NONEMPTY //only got here if nothing was accepted return EMPTY Q: Running time? Improvements?

35. EDFA A: Running time: O(|S||Q|) –exponential. Improvements: Just see if any accept state can be reached from start state by performing a BFS or DFS as follows:

36. EDFA DFAempty2( DFA M ) State q = q0 Stack S // Initialize a LIFO stack Set V = // Set of visited states S.push(q0) while(S  ) q = S.pop() if (qF ) return NONEMPTY V = V  {q} for (States q’ satisfying qq’ ) // an edge if (q’ V ) S.push(q’ ) return EMPTY // Only got here if F unreachable Q: Is this a BFS or DFS?

37. EDFA A: This is a DFS algorithm since stack is used. Queue would give BFS.

38. ACFG Acceptance problem is also solvable. There’s a rather horrible algorithm involving Chomsky Normal Form. Later we’ll see a much improved polynomial time algorithm (must have one, otherwise compilers would be useless!) Chomsky normal form gives the following fact: LEMMA: Suppose a grammar G is in Chomsky normal form. Let x be in L(G ), and let n be the length of x. Then x is generated by a derivation of length 2n-1, when n > 0.

39. ACFG LEMMA: Suppose a grammar G is in Chomsky normal form. Let x be in L(G ), and let n be the length of x. Then x is generated by a derivation of length 2n-1, when n > 0. Proof. The first n-1 productions are of the form A BC and get us to the correct length. The last n production are unit terminating A  a and derive x. This gives rise to the following algorithm:

40. ACFG CFGaccept( CFG G, String x=x1 x2 x3 … xn ) CFG G’ = ChomskyNormalForm(G ) for (all derivations from start variable of G’ of length  2n +1) if (derivation resulted in x) return ACCEPT return REJECT Q1: Why does this work for x =e? Q2: Running time in terms of n ?

41. ACFG A1: Works for x =e because of “ 2n+1” clause. A2: Exponential. If we consider the blow-up of CNF conversion, this is a truly horrendous algorithm.

42. Blackboard Exercises • EQDFA • ECFG • CARDDFA