New York University Theory of Computation Questions

User Generated

yrbFzvgu

Computer Science

ECS 120

New York University

ECS

Description

MCQs computational theory

MCQs computational theory MCQs computational theory

MCQs computational theory

MCQs computational theory

Unformatted Attachment Preview

Question 11 pts Mark all statements that have been proven to be true. Group of answer choices a. SAT is not decidable in polynomial time. b. SAT is decidable in polynomial time. c. P ⊆ NP d. P ≠ NP e. If SAT is decidable in polynomial time, then P = NP. f. If SAT is not decidable in polynomial time, then P ≠ NP. Flag question: Question 2 Question 21 pts Let A be a decision problem. Mark all that have been proven to be true. a. Group of answer choices b. If A is decidable by a polynomial-time nondeterministic algorithm, then it has a polynomial-time verifier. c. If A has a polynomial-time verifier, then it is decidable by a polynomial-time deterministic algorithm. d. If A is decidable by a polynomial-time deterministic algorithm, then it has a polynomial-time verifier. e. If A has a polynomial-time verifier, then it is decidable by a polynomial-time nondeterministic algorithm. Flag question: Question 3 Question 31 pts Mark all statements that are potentially true, meaning they are consistent with what we currently understand. In other words, unmark the statements that are provably false, and mark all the rest. Group of answer choices a. SAT, HamPath, SubsetSum, and Clique all have polynomial-time algorithms. b. The fastest algorithms for SAT, HamPath, SubsetSum, and Clique all require exponential time. c. SAT and HamPath have polynomial-time algorithms, but the fastest algorithms for SubsetSum and Clique require exponential time. d. SubsetSum and Clique have polynomial-time algorithms, but the fastest algorithms for SAT and HamPath require exponential time. Theory of Computation ECS 120 Lecture Notes David Doty ii Copyright © May 12, 2021, David Doty No part of this document may be reproduced without the expressed written consent of the author. All rights reserved. Contents 1 Introduction 1.1 What this course is about . . . . . . . . . . . 1.2 Automata, computability, and complexity . . 1.3 Mathematical background . . . . . . . . . . . 1.3.1 Implication statements . . . . . . . . . 1.3.2 Sets . . . . . . . . . . . . . . . . . . . 1.3.3 Sequences and tuples . . . . . . . . . . 1.3.4 Functions and relations . . . . . . . . . 1.3.5 The pigeonhole principle . . . . . . . . 1.3.6 Combinatorics . . . . . . . . . . . . . . 1.3.7 Graphs . . . . . . . . . . . . . . . . . . 1.3.8 Boolean logic . . . . . . . . . . . . . . 1.4 Proof by induction . . . . . . . . . . . . . . . 1.4.1 Proof by induction on natural numbers 1.4.2 Induction on other structures . . . . . I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Automata Theory 1 1 3 6 6 6 7 8 9 9 11 11 11 11 12 15 2 String theory 2.1 Why to study automata theory . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Binary numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 17 18 21 3 Deterministic finite automata 3.1 Intuitive overview of deterministic finite automata (DFA) 3.2 Formal models of computation . . . . . . . . . . . . . . . 3.3 Formal definition of a DFA (syntax) . . . . . . . . . . . . 3.4 More examples . . . . . . . . . . . . . . . . . . . . . . . 3.5 Formal definition of computation by a DFA (semantics) . 23 23 24 25 27 29 iii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv CONTENTS 4 Declarative models of computation 4.1 Regular expressions . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Formal definition of a regex (syntax and semantics) . . . 4.1.3 Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Context-free grammars . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Formal definition of a CFG (syntax) . . . . . . . . . . . 4.2.3 Formal definition of computation by a CFG (semantics) . 4.2.4 Right-regular grammars (RRG) . . . . . . . . . . . . . . 4.3 Nondeterministic finite automata (NFA) . . . . . . . . . . . . . 4.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Formal definition of an NFA (syntax) . . . . . . . . . . . 4.3.3 Set of states reachable by an NFA . . . . . . . . . . . . . 4.3.4 Transition function versus set of transitions . . . . . . . 4.3.5 Formal definition of computation by an NFA (semantics) 4.3.6 Example NFA using ε-transitions . . . . . . . . . . . . . 5 Closure of language classes under set operations 5.1 Automatic transformation of regex’s, NFAs, DFAs . 5.2 DFA union and intersection (product construction) 5.3 NFA union, concatenation, and star constructions . 5.3.1 Examples . . . . . . . . . . . . . . . . . . . 5.3.2 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 31 31 32 32 33 35 35 37 37 38 38 38 39 40 41 41 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 43 47 50 50 52 6 Equivalence of models 6.1 Equivalence of DFAs and NFAs (subset construction) . . . . . . 6.2 Equivalence of RGs and NFAs . . . . . . . . . . . . . . . . . . . 6.2.1 Left-regular grammars . . . . . . . . . . . . . . . . . . . 6.3 Equivalence of regex’s and NFAs . . . . . . . . . . . . . . . . . 6.3.1 Every regex-decidable language is NFA-decidable . . . . 6.3.2 Every NFA-decidable language is regex-decidable . . . . 6.4 Optional: Equivalence of DFAs and constant memory programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 57 60 62 63 63 64 68 . . . . . . 71 71 71 71 73 73 74 7 Proving problems are not solvable in a model of 7.1 Some languages are not regular . . . . . . . . . . 7.1.1 Why we need rigor . . . . . . . . . . . . . 7.1.2 A non-regular language . . . . . . . . . . . 7.1.3 Another non-regular language . . . . . . . 7.2 The pumping lemma for regular languages . . . . 7.3 Using the Pumping Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CONTENTS 7.4 7.5 7.6 II v Optional: Proof of the Pumping Lemma for regular languages . . . . . . . Optional: The Myhill-Nerode Theorem . . . . . . . . . . . . . . . . . . . . 7.5.1 Distinguishing extensions and statement of Myhill-Nerode Theorem 7.5.2 Examples of using Myhill-Nerode Theorem . . . . . . . . . . . . . . 7.5.3 Optional: Proof of Myhill-Nerode Theorem . . . . . . . . . . . . . . Optional: The Pumping Lemma for context-free languages . . . . . . . . . . . . . . . Computational Complexity and Computability Theory 8 Turing machines 8.1 Intuitive idea of Turing machines . . . . . . . . . . . . . . . . . . . . . 8.2 Formal definition of a TM (syntax) . . . . . . . . . . . . . . . . . . . . 8.3 Formal definition of computation by a TM (infinite semantics) . . . . . 8.4 Optional: Formal definition of computation by a TM (finite semantics) 8.5 Languages recognized/decided by TMs . . . . . . . . . . . . . . . . . . 8.6 Variants of TMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 Multitape TMs . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.2 Other variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 TMs versus code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Optional: The Church-Turing Thesis . . . . . . . . . . . . . . . . . . . 77 78 79 79 80 80 83 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 85 87 88 89 90 91 91 94 95 96 9 Efficient solution of problems: The class P 99 9.1 Asymptotic analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 9.1.1 Defining running time . . . . . . . . . . . . . . . . . . . . . . . . . . 99 9.1.2 Asymptotic Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 9.1.3 Rules of thumb for comparing growth rates . . . . . . . . . . . . . . . 101 9.1.4 Other asymptotic notations . . . . . . . . . . . . . . . . . . . . . . . 104 9.2 Time complexity classes and the Time Hierarchy Theorem . . . . . . . . . . 104 9.3 Optional: Time complexity of simulation of multitape TM with a one-tape TM105 9.4 Definition of P . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 9.4.1 The complexity class P. . . . . . . . . . . . . . . . . . . . . . . . . . 106 9.4.2 “Reasonable” encodings. . . . . . . . . . . . . . . . . . . . . . . . . . 106 9.4.3 Input size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 9.4.4 P is the same for most encodings and programming languages. . . . . 107 9.5 Examples of problems in P . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 9.5.1 Paths in graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 9.5.2 Relatively prime integers . . . . . . . . . . . . . . . . . . . . . . . . . 111 9.5.3 Connected graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 9.5.4 Optional: Eulerian cycles in graphs . . . . . . . . . . . . . . . . . . . 112 9.6 Optional: Why identify P with “efficient”? . . . . . . . . . . . . . . . . . . . 114 vi CONTENTS 10 Efficient verification of solutions: The class NP 10.1 Polynomial-time verifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.1 Hamiltonian path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.2 Composite numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 The class NP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Definition of NP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.2 What sort of computation is NP capturing? . . . . . . . . . . . . . . 10.2.3 Decision vs. Search vs. Optimization . . . . . . . . . . . . . . . . . . 10.3 Examples of problems in NP . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 Finding cliques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Finding subset of integers with a given sum . . . . . . . . . . . . . . 10.3.3 The P versus NP question . . . . . . . . . . . . . . . . . . . . . . . . 10.4 NP problems are decidable in exponential time . . . . . . . . . . . . . . . . . 10.5 Introduction to NP-Completeness . . . . . . . . . . . . . . . . . . . . . . . . 10.5.1 Boolean formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.2 Implementation of Boolean formula data structure in Python . . . . . 10.5.3 Satisfiability of Boolean formulas . . . . . . . . . . . . . . . . . . . . 10.6 Polynomial-time reducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.1 Ranking the hardness of problems with reductions . . . . . . . . . . . 10.6.2 Simple example: Reduction of IndSet to Clique . . . . . . . . . . . 10.6.3 Definition of polynomial-time reducibility . . . . . . . . . . . . . . . . 10.6.4 Using reductions to bound “hardness” of problems . . . . . . . . . . . 10.6.5 How to remember which direction reductions go . . . . . . . . . . . . 10.6.6 Definition of the 3Sat problem . . . . . . . . . . . . . . . . . . . . . 10.6.7 Reduction between problems with different data types: 3Sat ≤P IndSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.8 Reductions are algorithms, but they don’t solve either problem. . . . 10.7 Definition of NP-completeness . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8 The Cook-Levin Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.9 Optional: Additional NP-Complete problems . . . . . . . . . . . . . . . . . . 10.9.1 Vertex Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.9.2 Subset Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.9.3 Hamiltonian path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.10Optional: Proof of the Cook-Levin Theorem . . . . . . . . . . . . . . . . . . 10.11Optional: A brief history of the P versus NP problem . . . . . . . . . . . . . 117 117 117 118 119 119 120 120 121 121 123 123 124 126 126 127 129 129 129 131 131 132 134 135 11 Undecidability 11.1 The Halting Problem . . . . . . . . . . . . . . . 11.1.1 Turing-recognizable but not decidable . . 11.1.2 Reducibility . . . . . . . . . . . . . . . . 11.2 Optional: Source code versus programs . . . . . 11.3 Undecidable problems about algorithm behavior 153 153 153 154 155 156 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 138 138 140 141 141 143 145 145 149 CONTENTS . . . . . . . . . . . . . . . . . 156 158 158 159 160 160 163 167 170 170 170 173 173 175 176 176 178 A Reading Python A.1 Installing Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Code from this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Tutorial on reading Python . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 179 179 180 11.4 11.5 11.6 11.7 11.8 11.3.1 No-input halting problem . . . . . . . . . . . . . . . . . . . . . . 11.3.2 Recipe for showing undecidability via reduction from Haltsε . . 11.3.3 Accepting a given string . . . . . . . . . . . . . . . . . . . . . . . 11.3.4 Accepting no inputs . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.5 Rejecting at least one string . . . . . . . . . . . . . . . . . . . . . How to spot undecidability at a glance . . . . . . . . . . . . . . . . . . . Optional: Enumerators as an alternative definition of Turing-recognizable 11.5.1 A non-Turing-recognizable language . . . . . . . . . . . . . . . . . Different sizes of infinity . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.1 Comparing sizes of sets using onto functions . . . . . . . . . . . . 11.6.2 Can one infinite set be larger than another? . . . . . . . . . . . . 11.6.3 Cantor’s diagonalization technique . . . . . . . . . . . . . . . . . 11.6.4 Countable versus uncountable sets . . . . . . . . . . . . . . . . . Showing the halting problem is undecidable with diagonalization . . . . . Optional: The far-reaching consequences of undecidability . . . . . . . . 11.8.1 Gödel’s Incompleteness Theorem . . . . . . . . . . . . . . . . . . 11.8.2 Prediction of physical systems . . . . . . . . . . . . . . . . . . . . vii . . . . . . . . . . . . . . . . . viii CONTENTS Chapter 1 Introduction 1.1 What this course is about What are the fundamental capabilities and limitations of computers? The following is a rough sketch of what to expect from this course: • In ECS 36a/b/c (introductory programming sequence), you programmed computers without studying them mathematically. • In ECS 20 (discrete math), you mathematically studied things that are not computers. • In ECS 120 (theory of computation), you will use the tools of ECS 20 to mathematically study computers. The fundamental premise of the theory of computation is that the computer on your desk—or in your pocket—obeys certain laws, and therefore, certain unbreakable limitations. We can reason by analogy with the laws of physics. Newton’s equations of motion tell us that each object with mass obeys certain rules that cannot be broken. For instance, an object cannot accelerate in the opposite direction in which force is being applied to it. Of course, nothing in the world is a rigid frictionless body of mass obeying all the idealized assumptions of classical mechanics... in other words, “the map is not the territory”. So in reality, Newton’s equations of motion do not exactly predict anything. But they are a useful abstraction of what real matter is like, and many things in the world are close enough to this abstraction that Newtonian predictions are reasonably accurate. I often think of the field of computer science outside of theory as being about proving what can be done with a computer... simply by doing it! Much of research in theoretical computer science is about proving what cannot be done with a computer. This can be more difficult, since you cannot simply cite your failure to invent an algorithm solving a problem to be a proof that the problem cannot be solved. But as we will see, certain important problems cannot be solved with any algorithm. We will draw no distinction between the idea of “formal proof” and more nebulous instructions such as “show your work”/“justify your answer”/“explain”. A “proof” of a 1 2 CHAPTER 1. INTRODUCTION claim is an argument that convinces an intelligent person who has never seen the claim before and cannot see why it is true with having it explained. It does not matter if the argument uses formal mathematical notation or not (though formal notation is briefer and more straightforward to make precise than English), or if it uses proof by induction or proof by contradiction or just a direct proof (though it is often easier to think in terms of induction or contradiction). What matters is that there are no holes or counter-arguments that can be thrown at the argument, and that every statement is precise and unambiguous. However, one effective technique to prove theorems (I do this in nearly all of my research papers) is to first give an informal “proof sketch”, intuitively explaining how the proof will go, but shorter than the proof and lacking in potentially confusing (but necessary) details. The proof is easier to read if one first reads the proof sketch, but the proof sketch by itself is not a proof. In fact, I would go so far as to say that the proof by itself is not a very effective proof either, since bare naked details and mathematical notation, without any intuition to help someone understand it, do not communicate why the theorem is true any better than the hand-waving proof sketch. Both are usually necessary to accomplish the goal of the proof: to help the reader understand why the theorem is true. Here’s an example of a formal theorem, and informal proof sketch, and a formal proof: Theorem 1.1.1. There are infinitely many prime numbers. Proof sketch. Intuitively, we show that for each finite set of prime numbers, we can find a larger prime number. We do this by doing multiplying together the existing primes and adding 1. This is a new prime because, if it were a multiple of an existing prime, it would be only one greater than a different multiple of that prime. This is a contradiction because multiples of integers bigger than 1 are more widely spaced. (For example, multiples of 3 are at least 3 apart: 3, 6, 9, 12, 15, . . .) Proof. Let S = {p1 , p2 , . . . , pk } ⊂ N be any finite set of primes. It suffices to show that we can find a prime not in S, implying that S cannot be all the primes. Since S is an arbitrary finite set of primes, this shows that the set of all prime numbers cannot be finite. Let m = p1 · p2 · . . . · pk and let n = m + 1. Since n is larger than any number in S, thus unequal to any of them, it suffices to show n is also prime. Suppose for the sake of contradiction that n is not prime; then it has a prime factor pi 6= n, where pi | n (i.e., pi divides n). Note that pi | m as well. Therefore n and m are different multiples of pi , so their difference n − m must be at least pi . But n − m = 1, which is smaller than any prime, a contradiction. Note that the proof sketch is a bit shorter, uses less formal notation, and skips some details. As a result, it is easy to read, but incomplete. The proof gives all the details, but the proof sketch is a “roadmap” helping to guide the reader through it. Sometimes when I write a paper, I don’t necessarily keep the proof sketch separate from the proof. Sometimes the first paragraph or two of the proof is the sketch. Other times, the sketch is more “woven” into the proof, especially for long proofs. This helps the reader stop to “come up for air” 1.2. AUTOMATA, COMPUTABILITY, AND COMPLEXITY 3 occasionally and remember the big picture that the proof is trying to show. (However, in this course, most proofs will not be very long, so won’t require this level of care.) In this course, in the interest of time, I will often give the intuitive proof sketch only verbally, and write only the formal details on the board. On your homework, however, you should explicitly write both, to make it easy for the TA to understand your proof and give you full credit. 1.2 Automata, computability, and complexity Multivariate polynomials √ Consider the polynomial equation x2 −y 2 = 2. Does this have a solution? Yes: x = 2, y = 0 What about an integer solution? No. x2 + 2xy − y 3 = 13. Does this have an integer solution? Yes: x = 3, y = 2. Equations involving multivariate polynomials which we ask for a integer solution are called Diophantine equations. Task A: write an algorithm that indicates whether a given multivariable polynomial equation has a real solution. Fact: Task A is possible. Task A0 : write an algorithm that indicates whether a given multivariable polynomial equation has an real integer solution. Fact: Task A0 is impossible.1 Paths touring a graph Does the graph G = (V, E) given in Figure 1.1 have a path that contains each edge exactly once? (Eulerian path) Task B: write an “efficient” algorithm that indicates if a graph G has a path visiting each edge exactly once Algorithm: Check whether G is connected (by breadth-first search) and every node has even degree. Task B 0 : write an “efficient” algorithm that indicates if a graph G has a path visiting each edge node exactly once Fact: Task B 0 is impossible (assuming P 6= NP). Counting A regular expression (regex) is an expression that matches some strings and not others. For example (0(0 ∪ 1 ∪ 2)∗ ) ∪ ((0 ∪ 2)∗ 1) 1 We could imagine trying “all” possible integer solutions, but if there is no integer solution, then we will be trying forever and the algorithm will not halt. 4 CHAPTER 1. INTRODUCTION Figure 1.1: “Königsberg graph”. Licensed under CC BY-SA 3.0 via Commons – https://upload. wikimedia.org/wikipedia/commons/5/5d/Konigsberg_bridges.png matches any string of digits that starts with a 0, followed by any number of 0’s, 1’s, and 2’s, or ends with a 1, preceded by any number of 0’s and 2’s. If x is a binary string and a a symbol, let #(a, x) be the number of a’s in x. For example, #(0, 001200121) = 4 and #(1, 001200121) = 3. Task C: write a regex that matches every ternary string x exactly when #(0, x) + #(1, x) = 3. Answer: 2∗ (0 ∪ 1)2∗ (0 ∪ 1)2∗ (0 ∪ 1)2∗ Task C 0 : write a regex that matches every ternary string x exactly when #(0, x) − #(1, x) = 3. Fact: Task C 0 is impossible Rough layout of this course The following are the units of this course: Computability theory (unit 3): What problems can algorithms solve? (finding real roots of multivariate polynomials, but not integer roots) Computational complexity theory (unit 2): What problems can algorithms solve efficiently? (finding paths visiting every edge, but not every node) Automata theory (unit 1): What problems can algorithms solve with “optimal” efficiency (constant space and “real time”, i.e., time = size of input)? (finding whether the sum of # of 0’s and # of 1’s equals some constant, but not the difference) 1.2. AUTOMATA, COMPUTABILITY, AND COMPLEXITY 5 Sorting these by increasing order of the power of the algorithms studied: 1, 2, 3. Historically, these were discovered in the order 3, 1, 2. It is most common in a theory course to cover them in order 1, 3, 2. We are going to cover them in order 1, 2, 3. The reason for swapping the traditional order of units 3 (computability) and 2 (computational complexity) is this: Most of computer science is about writing programs to solve problems. You think about a problem, you write a program, and you demonstrate that the program solves the problem.2 Now, in computability theory, unit 3, most of the problems studied are questions about the behavior of programs themselves.3 But what sort of objects might answer these questions about programs? Other programs! So we imagine writing a program P that takes as input the source code of another program Q. This isn’t such a crazy idea yet. . . compilers, interpreters (such as the Javascript engine in a web browser), and virtual machines are programs that take as input the source code of other programs. But in computability theory, the situation complicates quickly. A typical problem involves proving that a certain problem is not solvable by any program. Usually this done by writing a program P , which takes as input another program Q, and P outputs a third program R.4 It can be quite taxing to keep track of which program is supposed to be doing what. So in the interest of showing no program exists to solve a certain problem, we introduce not one but three new programs, all of which are not solving that problem. It’s quite remarkable that anyone was able to create such a chain of reasoning in the first place to prove limits on the ability of programs. It adds many conceptual layers of abstraction onto what most computer science students are accustomed to doing. On the other hand, computational complexity theory, unit 2, has the virtue that most problems are about more mundane objects such as lists of integers, graphs, and Boolean formulas. In complexity theory, we think about programs that take these objects as input and produce them as output, and there is less danger of getting the program confused with a graph, for instance, because they are two different types of objects. The ideas in units 2 and 3 are more similar to each other than they are to unit 1. Also, we will spend about half the course on unit 1, and the other half on units 2 and 3. So, these notes are divided into two “parts”: part 1 is unit 1, automata theory, and part 2 is units 2 and 3, computational complexity and computability theory. 2 In algorithms and theory courses and research, to “demonstrate correctness” usually means a proof. In software engineering, it usually means unit tests and code reviews. For critical stuff like spacecraft software, it involves both. 3 The Diophantine equation problem above is a notable exception, but it took 70 years to prove it is unsolvable, and even then it was done by creating a Diophantine equation that essentially “mimics” the behavior of a program so as to connect the existence of its roots to a question about the behavior of certain programs. 4 Now, this sort of idea, of programs receiving and producing other programs, is not crazy in principle. The C compiler, for example, is itself a program, which takes as input the source code of a program (written in C) and outputs the code of another program (machine instructions, written in the “language” of the host machine’s instruction set). The C preprocessor (which rewrites macros, for instance) takes as input C programs and produces C programs as output. 6 CHAPTER 1. INTRODUCTION 1.3 1.3.1 Mathematical background Implication statements Given two boolean statements p and q 5 , the implication p =⇒ q is shorthand for “p implies q”, or “If p is true, then q is true” 6 , p is the hypothesis, and q is the conclusion. The following statements are related to p =⇒ q: • the inverse: ¬p =⇒ ¬q • the converse: q =⇒ p • the contrapositive: ¬q =⇒ ¬p7 If an implication statement p =⇒ q and its converse q =⇒ p are both true, then we say p if and only if (iff) q, written p ⇐⇒ q. Proving a “p ⇐⇒ q” theorem usually involves proving p =⇒ q and q =⇒ p separately. 1.3.2 Sets A set is a group of objects, called elements, with no duplicates.8 The cardinality of a set A is the number of elements it contains, written |A|. For example, {7, 21, 57} is the set consisting of the integers 7, 21, and 57, with cardinality 3. For two sets A and B, we write A ⊆ B, and say that A is a subset of B, if every element of A is also an element of B. A is a proper subset of B, written A B, if A ⊆ B and A 6= B. We use the following sets throughout the course • the natural numbers N = {0, 1, 2, . . .} • the integers Z = {. . . , −2, −1, 0, 1, 2, . . .}   p • the rational numbers Q = p ∈ Z, q ∈ N, q 6= 0 q • the real numbers R The unique set with no elements is called the empty set, written ∅. To define sets symbolically,9 we use set-builder notation: for instance, { x ∈ N | x is odd } is the set of all odd natural numbers. 5 e.g., “Hawaii is west of California”, or “The stoplight is green.” e.g., “If the stoplight is green, then my car can go.” 7 The contrapositive of a statement is logically equivalent to the statement itself. For example, it is equivalent to state “If someone is allowed to drink alcohol, then they are at least 21” and “If someone is under 21, then they are not allowed drink alcohol”. Hence a statement’s converse and inverse are logically equivalent to each other, though not equivalent to the statement itself. 8 Think of std::set. 9 In other words, to express them without listing all of their elements explicitly, which is convenient for large finite sets and necessary for infinite sets. 6 1.3. MATHEMATICAL BACKGROUND 7 We write ∀x ∈ A as a shorthand for “for all elements x in the set A ...”, and ∃x ∈ A as a shorthand for “there exists an element x in the set A such that ...”. For example, (∃n ∈ N) n > 10 means “there exists a natural number n such that n is greater than 10”. Given two sets A and B, A∪B = { x | x ∈ A or x ∈ B } is the union of A and B, A∩B = { x | x ∈ A and x ∈ B } is the intersection of A and B, and A \ B = { x ∈ A | x 6∈ B } is the difference between A and B (also written A − B). A = { x | x 6∈ A } is the complement of A. 10 Given a set A, P(A) = { S | S ⊆ A } is the power set of A, the set of all subsets of A. For example, P({2, 3, 5}) = {∅, {2}, {3}, {5}, {2, 3}, {2, 5}, {3, 5}, {2, 3, 5}}. Given any set A, it always holds that ∅, A ∈ P(A), and that |P(A)| = 2|A| if |A| < ∞. 1.3.3 11 12 Sequences and tuples A sequence is an ordered list of objects 13 . For example, (7, 21, 57, 21) is the sequence of integers 7, then 21, then 57, then 21. A tuple is a finite sequence.14 (7, 21, 57) is a 3-tuple. A 2-tuple is called a pair. For two sets A and B, the cross product of A and B is A×B = { (a, b) | a ∈ A and b ∈ B }. Sk i ≤k Note that |A×B| = |A|·|B|. For k ∈ N, we write Ak = A × A × . . . × A and A = i=0 A . | {z } k times For example, N2 = N × N is the set of all ordered pairs of natural numbers. 10 Usually, if A is understood to be a subset of some larger set U , the “universe” of possible elements, then A is understood to be U \ A. For example if we are dealing only with N, and A ⊆ N, then A = { n ∈ N | n 6∈ A }. In other words, we used “typed” sets, in which case each set we use has some unique superset – such as {0, 1}∗ , N, R, Q, the set of all finite automata, etc. – that is considered to contain all the elements of the same type as the elements of the set we are discussing. Otherwise, we would have the awkward situation that for A ⊆ N, A would contain not only nonnegative integers that are not in A, but also negative integers, real numbers, strings, functions, stuffed animals, and other objects that are not elements of A. 11 Why? 12 Actually, Cantor’s theory of infinite set cardinalities makes sense of the claim that |P(A)| = 2|A| even if A is an infinite set. The furthest we will study this theory in this course is to observe that there are at least two infinite set cardinalities: that of the set of natural numbers, and that of the set of real numbers, which is bigger than the set of natural numbers according to this theory. 13 Think of std::vector. 14 The closest C++ analogy to a tuple, as we will use them in this course, is an object. Each member variable of an object is like an element of the tuple, although C++ is different in that each member variable of an object has a name, whereas the only way to distinguish one element of a tuple from another is their position. But when we use tuples, for instance to define a finite automaton as a 5-tuple, we intuitively think of the 5 elements as being like 5 member variables that would be used to define a finite automaton object. And of course, the natural way to implement such an object in C++ by defining a FiniteAutomaton class with 5 member variables, which is an easier way to keep track of what each of the 5 elements is supposed to represent than, for instance, using an void[] array of length 5. 8 CHAPTER 1. INTRODUCTION 1.3.4 Functions and relations A function f that takes an input from set D (the domain) and produces an output in set R (the range) is written f : D → R. 15 Given A ⊆ D, define f (A) = { f (x) | x ∈ A }; call this the image of A under f . Given f : D → D, k ∈ N and d ∈ D, define f k : D → D by f k (d) = f (f (. . . f ( d)) . . .)) | {z } k times to be f composed with itself k times. If f might not be defined for some values in the domain, we say f is a partial function.16 If f is defined on all values, it is a total function.17 A function f with a finite domain can be represented with a table. For example, the function f : {0, 1, 2, 3} → Q defined by f (n) = n2 is represented by the table n f (n) 0 0 1 1 2 2 1 3 3 2 If (∀d1 , d2 ∈ D) d1 6= d2 =⇒ f (d1 ) 6= f (d2 ), then we say f is 1-1 (one-to-one or injective).18 If (∀r ∈ R)(∃d ∈ D) f (d) = r, then we say f is onto (surjective). Intuitively, f “covers” the range R, in the sense that no element of R is left un-mapped-to by f . f is a bijection (a.k.a. a 1-1 correspondence) if f is both 1-1 and onto. A predicate is a function whose output is boolean. Given a set A, a relation R on A is a subset of A × A. Intuitively, the elements in R are the ones related to each other. Relations are often written with an operator; for instance, the relation ≤ on N is the set R = { (n, m) ∈ N × N | (∃k ∈ N) n + k = m }. 15 Most statically typed programming languages like C++ have direct support for functions with declared types for input and output. In Java, these are like static methods; Integer.parseInt, which takes a String and returns the int that the String represents (if it indeed represents an integer) is like a function with domain String and range int. Math.max is like a function with domain int × int (since it accepts a pair of ints as input) and range int. The main difference between functions in programming languages and those in mathematics is that in a programming language, a function is really an algorithm for computing the output, given in the input, whereas in mathematics the function is just the abstract relationship between input and output, and there may not be any algorithm computing it. 16 For instance, Integer.parseInt is (strictly) partial, because not all Strings look like integers, and such Strings will cause the method to throw a NumberFormatException. 17 Every total function is a partial function, but the converse does not hold for any function that is undefined for at least one value. We will usually assume that functions are total unless explicitly stated otherwise. 18 Intuitively, f does not map any two points in D to the same point in R. It does not lose information; knowing an output r ∈ R suffices to identify the input d ∈ D that produced it (through f ). 1.3. MATHEMATICAL BACKGROUND 1.3.5 9 The pigeonhole principle The pigeonhole principle is a very simple concept, but it is surprisingly powerful. It says that if we put n objects a1 , a2 , . . . , an into fewer than n boxes, then there is a box with at least two objects. Usually, we think of the boxes as being “properties” or “labels” the objects have: if there are n objects but fewer than n labels, then two objects must get the same label. For example, all people have fewer than 200,000 hairs on their head. There are about 500,000 people in Sacramento. If we think of the people as the objects a1 , a2 , . . . , a500,000 , and we think of the count of hairs as the labels (perhaps person a1 has 107,235 hairs, person a2 has 95,300 hairs, etc.) then since there are only 200,000 labels 0, 1, 2, . . . ,199,999, at least two people must have the exact same number of hairs on their head. Here’s a way of stating the pigeonhole principle using functions. If I have a set O of objects and a set B of boxes, and if |O| > |B|, then every function f : O → B is not 1-1. In other words, for every function f : O → B, there are two different objects o1 , o2 ∈ O such that f (o1 ) = f (o2 ), i.e., f puts o1 and o2 in the same box.19 1.3.6 Combinatorics Counting sizes of sets is a tricky art to learn. It’s a whole subfield of mathematics called combinatorics, with very deep theorems, but we won’t need anything particularly deep in this course. For simple counting problems, a few basic principles apply. The first is sometimes called “The Product Rule”: If a set is defined by cross-product (i.e., each element of a set is a tuple), then often multiplying the sizes of the smaller sets works. For example, there are 4 integers in the set A = {1, 2, 3, 4} and 3 integers in the set B = {1, 2, 3}. How many ways are there to pick one element from A and one from B? There are 12 pairs of integers in the set A × B, because to choose a pair (a, b) ∈ A × B, there are 4 ways to choose a and 3 ways to choose b, so 4 · 3 ways to choose both. Here they all are: 1 2 3 4 1 (1, 1) (2, 1) (3, 1) (4, 1) 2 (1, 2) (2, 2) (3, 2) (4, 2) 3 (1, 3) (2, 3) (3, 3) (4, 3) If the tuple is bigger, you just keep multiplying: in the triple (a, b, c), if there are 4 ways to choose a, 3 ways to choose b, and 7 ways to choose c, then there are 4 · 3 · 7 = 84 possible triples (a, b, c). 19 In fact, we will eventually see (Section 11.6.3) that such reasoning applies even to infinite sets. If O and B are both infinite, but there is no 1-1 function f : O → B, then any way of assigning objects from O to “boxes” in B must assign two objects to the same box. For example, we will see that if O is the set of all decision problems (defined formally in Section 2.2) we want to solve with algorithms (defined formally as Turing machines in Chapter 8), and B is the set of all algorithms, then |O| > |B|, so there is no 1-1 function f : O → B. So any way of assigning each decision problem uniquely to an algorithm that solves it must fail (by assigning two different problems to the same algorithm, which can’t very well solve both of them). Thus some decision problems have no algorithm solving them. 10 CHAPTER 1. INTRODUCTION As another example, what’s the number of functions f : {1, 2, . . . , n} → {1, 2, . . . , k}? There are k ways to choose what f (1) is, times k ways to choose what f (2) is, etc., and there are n such values to choose. So there are k · . . . · k} = k n ways to choose f . | · k {z n The key here is that the element from set is “chosen independently”, meaning it’s possible to have any combination of them together, with no extra constraints. Sometimes this doesn’t hold. Suppose I define a new set C, not equal to A × B, but defined as the set of all pairs (a, b), where a ∈ A, b ∈ B, and a ≤ b. Then C is not equal to A × B, but instead C is a strict subset of A × B (meaning there are extra constraints applied so that not all possible pairs in A × B are allowed in C), because of the last condition that a ≤ b, so you cannot choose a and b independently. Here is that set (with 6 elements) for A = {1, 2, 3, 4} and B = {1, 2, 3}: 1 2 3 1 (1, 1) (1, 2) (1, 3) 2 (2, 2) (2, 3) 3 (3, 3) 4 In the function example, if we had asked what’s the number of non-decreasing functions f : {1, 2, . . . , n} → {1, 2, . . . , k}, then the choices are not independent: If we choose f (1) = 5, then we could not choose f (2) = 3. The other basic tools that get used are: • There are n! = n · (n − 1) · (n − 2) · . . . · 2 · 1 permutations of a sequence of n elements. For example, the 3! = 3 · 2 · 1 = 6 permutations of {1, 2, 3} are 123, 132, 213, 231, 312, 321.  n·(n−1)·(k+2)·(k+1) n! • There are nk = k!(n−k)! = (n−k)·(n−k−1)·...·2·1 ways to choose a set of k elements from a larger set of n elements. For example, the number of binary strings of length 5 with  5·(5−1) 5 = 10: that’s the number of ways to pick 2 positions exactly two 1’s is 2 = 2 in the string to be 1, out of 5 possible positions: 00011, 00101, 01001, 10001, 00110, 01010, 10010, 01100, 10100, 11000. • Split up into sub-cases that can be added together. The next part uses this idea. • Watch carefully for parts that are dependent, to try to phrase how to choose them in a way that involves only independent choices. For example, suppose I ask how many 4-digit integers have the same first and last digit (e.g., 1471, 5015, 3223). If I say “there are 9 ways to choose the first digit, times 10 ways to choose the second, times 10 ways to choose the third, times 10 ways to choose the fourth, but there is some tricky dependence with the first and fourth”, it’s not clear what to do. But we can phrase it like this: there are 9 · 10 · 10 = 900 ways to choose the first three digits, and once I’ve picked the first digit, there is no choice with the fourth. So there are 900 total such numbers. 1.4. PROOF BY INDUCTION 11 A slightly trickier example is (where we analyze two different sub-cases): how many 4 digit numbers have their first digit within 1 of their last digit? (for example: 1230, 1231, 1232, 9009, 9008) Well, there’s still 900 ways to choose the first three digits, but there’s now 3 ways to choose the last digit, unless the first digit is 9, and then there’s only 2 ways to choose the last digit. This gives two sub-cases: the first digit is 9 or it isn’t. There are 800 ways to choose the first three digits with the first not equal to 9, times 3 ways to choose the last digit (800 · 3 = 2400). There are 100 ways to choose the first three digits with the first equal to 9, times 2 ways to choose the last digit (100·2 = 200). Combining the sub-cases, there are 2400 + 200 = 2600 such numbers. 1.3.7 Graphs See slides. 1.3.8 Boolean logic See slides. 1.4 Proof by induction Proof by induction is a potentially confusing concept, seeming a bit mysterious compared to direct proofs or proofs by contradiction. Luckily, you already understand proof by induction better than most people, since it is merely the “proof” version of the technique of recursion you learned in programming courses. 1.4.1 Proof by induction on natural numbers Theorem 1.4.1. For every n ∈ N, |{0, 1}n | = 2n . Proof. (by induction on n) 20 Base case: {0, 1}0 = {ε}.21 |{ε}| = 1 = 20 , so the base case holds. Inductive case: Assume |{0, 1}n−1 | = 2n−1 .22 We must prove that |{0, 1}n | = 2n . Note that every x ∈ {0, 1}n−1 appears as a prefix of exactly two unique strings in {0, 1}n , 20 To start, state in English what the theorem is saying: For every string length n, there are 2n strings of length n. Note that {0, 1}0 is not ∅; there is always one string of length 0, so the set of such strings is not empty. 22 Call this the inductive hypothesis, the fact we get to assume is true in proving the inductive case. 21 12 CHAPTER 1. INTRODUCTION namely x0 and x1.23 Then |{0, 1}n | =2 · |{0, 1}n−1 | =2 · 2n−1 =2n . Of course, there are other (non-induction) using the Product Rule for counting, we can bit, times 2 ways to choose the second bit, 2| · 2 ·{z. . . · 2} = 2n ways to choose all of them. inductive hypothesis ways to see that |{0, 1}n | = 2n . For example, say that there are 2 ways to choose the first ..., times 2 ways to choose the last bit, so This is a sort of “iterative” reasoning that is n more cleanly (but also more verbosely and pedantically) captured by the inductive argument above. P 1 n Theorem 1.4.2. For every n ∈ N+ , ni=1 i(i+1) = n+1 . Pn 1 1 1 n Proof. Base case (n = 1): i=1 i(i+1) = 1(1+1) = 2 = n+1 , so the base case holds. Inductive case: Let n ∈ N+ and suppose the theorem holds for n. Then n+1 X i=1 n X 1 1 1 = + i(i + 1) (n + 1)(n + 2) i=1 i(i + 1) 1 n + (n + 1)(n + 2) n + 1 1 + n(n + 2) = (n + 1)(n + 2) n2 + 2n + 1 = (n + 1)(n + 2) (n + 1)2 = (n + 1)(n + 2) n+1 = , n+2 = pull out last term inductive hypothesis so the inductive case holds. 1.4.2 Induction on other structures Induction is often taught as something that applies only to natural numbers, but one can write recursive algorithms that operate on data structures other than natural numbers. 23 The fact that they are unique means that if we count two strings in {0, 1}n for every one string in {0, 1}n−1 , we won’t double-count any strings. Hence |{0, 1}n | = 2 · |{0, 1}n−1 | 1.4. PROOF BY INDUCTION 13 Similarly, it is possible to prove something by induction on something other than a natural number. Here is an inductive definition of the number of 0’s in a binary string x, denoted #(0, x).24  if x = ε; (base case)  0, ∗ #(0, w) + 1, if x = w0 for some w ∈ {0, 1} ; (inductive case) #(0, x) =  #(0, w), if x = w1 for some w ∈ {0, 1}∗ . (inductive case) To prove a theorem by induction, identify the base case as the “smallest” object25 for which the theorem holds.26 Theorem 1.4.3. Every binary tree T of depth d has at most 2d leaves. Proof. (by induction on a binary tree T ) For T a tree, let d(T ) be the depth of T , and l(T ) the number of leaves in T . Base case: Let T be the tree with one node. Then d(T ) = 0, and 20 = 1 = l(T ). Inductive case: Let T ’s root have subtrees T0 and T1 , at least one of them non-empty. If only one is non-empty (say Ti ), then l(T ) =l(Ti ) ≤2d(Ti ) inductive hypothesis =2d(T )−1 definition of depth d(T )
Purchase answer to see full attachment
User generated content is uploaded by users for the purposes of learning and should be used following Studypool's honor code & terms of service.

Explanation & Answer

View attached explanation and answer. Let me know if you have any questions.

Question 11 pts
Mark all statements that have been proven to be true.
Group of answer choices
a. SAT is not decidable in polynomial time.
b. SAT is decidable in polynomial time.
c. P ⊆ NP
d. P ≠ NP
e. If SAT is decidable in polynomial time, then P = NP.
f. If SAT is ...


Anonymous
Just what I needed…Fantastic!

Studypool
4.7
Trustpilot
4.5
Sitejabber
4.4

Related Tags