Introduction to Theoretical Computer Science — Boaz Barak

Index

\[ \newcommand{\undefined}{} \newcommand{\hfill}{} \newcommand{\qedhere}{\square} \newcommand{\qed}{\square} \newcommand{\ensuremath}[1]{#1} \newcommand{\bbA}{\mathbb A} \newcommand{\bbB}{\mathbb B} \newcommand{\bbC}{\mathbb C} \newcommand{\bbD}{\mathbb D} \newcommand{\bbE}{\mathbb E} \newcommand{\bbF}{\mathbb F} \newcommand{\bbG}{\mathbb G} \newcommand{\bbH}{\mathbb H} \newcommand{\bbI}{\mathbb I} \newcommand{\bbJ}{\mathbb J} \newcommand{\bbK}{\mathbb K} \newcommand{\bbL}{\mathbb L} \newcommand{\bbM}{\mathbb M} \newcommand{\bbN}{\mathbb N} \newcommand{\bbO}{\mathbb O} \newcommand{\bbP}{\mathbb P} \newcommand{\bbQ}{\mathbb Q} \newcommand{\bbR}{\mathbb R} \newcommand{\bbS}{\mathbb S} \newcommand{\bbT}{\mathbb T} \newcommand{\bbU}{\mathbb U} \newcommand{\bbV}{\mathbb V} \newcommand{\bbW}{\mathbb W} \newcommand{\bbX}{\mathbb X} \newcommand{\bbY}{\mathbb Y} \newcommand{\bbZ}{\mathbb Z} \newcommand{\sA}{\mathscr A} \newcommand{\sB}{\mathscr B} \newcommand{\sC}{\mathscr C} \newcommand{\sD}{\mathscr D} \newcommand{\sE}{\mathscr E} \newcommand{\sF}{\mathscr F} \newcommand{\sG}{\mathscr G} \newcommand{\sH}{\mathscr H} \newcommand{\sI}{\mathscr I} \newcommand{\sJ}{\mathscr J} \newcommand{\sK}{\mathscr K} \newcommand{\sL}{\mathscr L} \newcommand{\sM}{\mathscr M} \newcommand{\sN}{\mathscr N} \newcommand{\sO}{\mathscr O} \newcommand{\sP}{\mathscr P} \newcommand{\sQ}{\mathscr Q} \newcommand{\sR}{\mathscr R} \newcommand{\sS}{\mathscr S} \newcommand{\sT}{\mathscr T} \newcommand{\sU}{\mathscr U} \newcommand{\sV}{\mathscr V} \newcommand{\sW}{\mathscr W} \newcommand{\sX}{\mathscr X} \newcommand{\sY}{\mathscr Y} \newcommand{\sZ}{\mathscr Z} \newcommand{\sfA}{\mathsf A} \newcommand{\sfB}{\mathsf B} \newcommand{\sfC}{\mathsf C} \newcommand{\sfD}{\mathsf D} \newcommand{\sfE}{\mathsf E} \newcommand{\sfF}{\mathsf F} \newcommand{\sfG}{\mathsf G} \newcommand{\sfH}{\mathsf H} \newcommand{\sfI}{\mathsf I} \newcommand{\sfJ}{\mathsf J} \newcommand{\sfK}{\mathsf K} \newcommand{\sfL}{\mathsf L} \newcommand{\sfM}{\mathsf M} \newcommand{\sfN}{\mathsf N} \newcommand{\sfO}{\mathsf O} \newcommand{\sfP}{\mathsf P} \newcommand{\sfQ}{\mathsf Q} \newcommand{\sfR}{\mathsf R} \newcommand{\sfS}{\mathsf S} \newcommand{\sfT}{\mathsf T} \newcommand{\sfU}{\mathsf U} \newcommand{\sfV}{\mathsf V} \newcommand{\sfW}{\mathsf W} \newcommand{\sfX}{\mathsf X} \newcommand{\sfY}{\mathsf Y} \newcommand{\sfZ}{\mathsf Z} \newcommand{\cA}{\mathcal A} \newcommand{\cB}{\mathcal B} \newcommand{\cC}{\mathcal C} \newcommand{\cD}{\mathcal D} \newcommand{\cE}{\mathcal E} \newcommand{\cF}{\mathcal F} \newcommand{\cG}{\mathcal G} \newcommand{\cH}{\mathcal H} \newcommand{\cI}{\mathcal I} \newcommand{\cJ}{\mathcal J} \newcommand{\cK}{\mathcal K} \newcommand{\cL}{\mathcal L} \newcommand{\cM}{\mathcal M} \newcommand{\cN}{\mathcal N} \newcommand{\cO}{\mathcal O} \newcommand{\cP}{\mathcal P} \newcommand{\cQ}{\mathcal Q} \newcommand{\cR}{\mathcal R} \newcommand{\cS}{\mathcal S} \newcommand{\cT}{\mathcal T} \newcommand{\cU}{\mathcal U} \newcommand{\cV}{\mathcal V} \newcommand{\cW}{\mathcal W} \newcommand{\cX}{\mathcal X} \newcommand{\cY}{\mathcal Y} \newcommand{\cZ}{\mathcal Z} \newcommand{\bfA}{\mathbf A} \newcommand{\bfB}{\mathbf B} \newcommand{\bfC}{\mathbf C} \newcommand{\bfD}{\mathbf D} \newcommand{\bfE}{\mathbf E} \newcommand{\bfF}{\mathbf F} \newcommand{\bfG}{\mathbf G} \newcommand{\bfH}{\mathbf H} \newcommand{\bfI}{\mathbf I} \newcommand{\bfJ}{\mathbf J} \newcommand{\bfK}{\mathbf K} \newcommand{\bfL}{\mathbf L} \newcommand{\bfM}{\mathbf M} \newcommand{\bfN}{\mathbf N} \newcommand{\bfO}{\mathbf O} \newcommand{\bfP}{\mathbf P} \newcommand{\bfQ}{\mathbf Q} \newcommand{\bfR}{\mathbf R} \newcommand{\bfS}{\mathbf S} \newcommand{\bfT}{\mathbf T} \newcommand{\bfU}{\mathbf U} \newcommand{\bfV}{\mathbf V} \newcommand{\bfW}{\mathbf W} \newcommand{\bfX}{\mathbf X} \newcommand{\bfY}{\mathbf Y} \newcommand{\bfZ}{\mathbf Z} \newcommand{\rmA}{\mathrm A} \newcommand{\rmB}{\mathrm B} \newcommand{\rmC}{\mathrm C} \newcommand{\rmD}{\mathrm D} \newcommand{\rmE}{\mathrm E} \newcommand{\rmF}{\mathrm F} \newcommand{\rmG}{\mathrm G} \newcommand{\rmH}{\mathrm H} \newcommand{\rmI}{\mathrm I} \newcommand{\rmJ}{\mathrm J} \newcommand{\rmK}{\mathrm K} \newcommand{\rmL}{\mathrm L} \newcommand{\rmM}{\mathrm M} \newcommand{\rmN}{\mathrm N} \newcommand{\rmO}{\mathrm O} \newcommand{\rmP}{\mathrm P} \newcommand{\rmQ}{\mathrm Q} \newcommand{\rmR}{\mathrm R} \newcommand{\rmS}{\mathrm S} \newcommand{\rmT}{\mathrm T} \newcommand{\rmU}{\mathrm U} \newcommand{\rmV}{\mathrm V} \newcommand{\rmW}{\mathrm W} \newcommand{\rmX}{\mathrm X} \newcommand{\rmY}{\mathrm Y} \newcommand{\rmZ}{\mathrm Z} \newcommand{\paren}[1]{( #1 )} \newcommand{\Paren}[1]{\left( #1 \right)} \newcommand{\bigparen}[1]{\bigl( #1 \bigr)} \newcommand{\Bigparen}[1]{\Bigl( #1 \Bigr)} \newcommand{\biggparen}[1]{\biggl( #1 \biggr)} \newcommand{\Biggparen}[1]{\Biggl( #1 \Biggr)} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\bigabs}[1]{\bigl\lvert #1 \bigr\rvert} \newcommand{\Bigabs}[1]{\Bigl\lvert #1 \Bigr\rvert} \newcommand{\biggabs}[1]{\biggl\lvert #1 \biggr\rvert} \newcommand{\Biggabs}[1]{\Biggl\lvert #1 \Biggr\rvert} \newcommand{\card}[1]{\lvert #1 \rvert} \newcommand{\Card}[1]{\left\lvert #1 \right\rvert} \newcommand{\bigcard}[1]{\bigl\lvert #1 \bigr\rvert} \newcommand{\Bigcard}[1]{\Bigl\lvert #1 \Bigr\rvert} \newcommand{\biggcard}[1]{\biggl\lvert #1 \biggr\rvert} \newcommand{\Biggcard}[1]{\Biggl\lvert #1 \Biggr\rvert} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\bignorm}[1]{\bigl\lVert #1 \bigr\rVert} \newcommand{\Bignorm}[1]{\Bigl\lVert #1 \Bigr\rVert} \newcommand{\biggnorm}[1]{\biggl\lVert #1 \biggr\rVert} \newcommand{\Biggnorm}[1]{\Biggl\lVert #1 \Biggr\rVert} \newcommand{\iprod}[1]{\langle #1 \rangle} \newcommand{\Iprod}[1]{\left\langle #1 \right\rangle} \newcommand{\bigiprod}[1]{\bigl\langle #1 \bigr\rangle} \newcommand{\Bigiprod}[1]{\Bigl\langle #1 \Bigr\rangle} \newcommand{\biggiprod}[1]{\biggl\langle #1 \biggr\rangle} \newcommand{\Biggiprod}[1]{\Biggl\langle #1 \Biggr\rangle} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\Set}[1]{\left\lbrace #1 \right\rbrace} \newcommand{\bigset}[1]{\bigl\lbrace #1 \bigr\rbrace} \newcommand{\Bigset}[1]{\Bigl\lbrace #1 \Bigr\rbrace} \newcommand{\biggset}[1]{\biggl\lbrace #1 \biggr\rbrace} \newcommand{\Biggset}[1]{\Biggl\lbrace #1 \Biggr\rbrace} \newcommand{\bracket}[1]{\lbrack #1 \rbrack} \newcommand{\Bracket}[1]{\left\lbrack #1 \right\rbrack} \newcommand{\bigbracket}[1]{\bigl\lbrack #1 \bigr\rbrack} \newcommand{\Bigbracket}[1]{\Bigl\lbrack #1 \Bigr\rbrack} \newcommand{\biggbracket}[1]{\biggl\lbrack #1 \biggr\rbrack} \newcommand{\Biggbracket}[1]{\Biggl\lbrack #1 \Biggr\rbrack} \newcommand{\ucorner}[1]{\ulcorner #1 \urcorner} \newcommand{\Ucorner}[1]{\left\ulcorner #1 \right\urcorner} \newcommand{\bigucorner}[1]{\bigl\ulcorner #1 \bigr\urcorner} \newcommand{\Bigucorner}[1]{\Bigl\ulcorner #1 \Bigr\urcorner} \newcommand{\biggucorner}[1]{\biggl\ulcorner #1 \biggr\urcorner} \newcommand{\Biggucorner}[1]{\Biggl\ulcorner #1 \Biggr\urcorner} \newcommand{\ceil}[1]{\lceil #1 \rceil} \newcommand{\Ceil}[1]{\left\lceil #1 \right\rceil} \newcommand{\bigceil}[1]{\bigl\lceil #1 \bigr\rceil} \newcommand{\Bigceil}[1]{\Bigl\lceil #1 \Bigr\rceil} \newcommand{\biggceil}[1]{\biggl\lceil #1 \biggr\rceil} \newcommand{\Biggceil}[1]{\Biggl\lceil #1 \Biggr\rceil} \newcommand{\floor}[1]{\lfloor #1 \rfloor} \newcommand{\Floor}[1]{\left\lfloor #1 \right\rfloor} \newcommand{\bigfloor}[1]{\bigl\lfloor #1 \bigr\rfloor} \newcommand{\Bigfloor}[1]{\Bigl\lfloor #1 \Bigr\rfloor} \newcommand{\biggfloor}[1]{\biggl\lfloor #1 \biggr\rfloor} \newcommand{\Biggfloor}[1]{\Biggl\lfloor #1 \Biggr\rfloor} \newcommand{\lcorner}[1]{\llcorner #1 \lrcorner} \newcommand{\Lcorner}[1]{\left\llcorner #1 \right\lrcorner} \newcommand{\biglcorner}[1]{\bigl\llcorner #1 \bigr\lrcorner} \newcommand{\Biglcorner}[1]{\Bigl\llcorner #1 \Bigr\lrcorner} \newcommand{\bigglcorner}[1]{\biggl\llcorner #1 \biggr\lrcorner} \newcommand{\Bigglcorner}[1]{\Biggl\llcorner #1 \Biggr\lrcorner} \newcommand{\expr}[1]{\langle #1 \rangle} \newcommand{\Expr}[1]{\left\langle #1 \right\rangle} \newcommand{\bigexpr}[1]{\bigl\langle #1 \bigr\rangle} \newcommand{\Bigexpr}[1]{\Bigl\langle #1 \Bigr\rangle} \newcommand{\biggexpr}[1]{\biggl\langle #1 \biggr\rangle} \newcommand{\Biggexpr}[1]{\Biggl\langle #1 \Biggr\rangle} \newcommand{\e}{\varepsilon} \newcommand{\eps}{\varepsilon} \newcommand{\from}{\colon} \newcommand{\super}[2]{#1^{(#2)}} \newcommand{\varsuper}[2]{#1^{\scriptscriptstyle (#2)}} \newcommand{\tensor}{\otimes} \newcommand{\eset}{\emptyset} \newcommand{\sse}{\subseteq} \newcommand{\sst}{\substack} \newcommand{\ot}{\otimes} \newcommand{\Esst}[1]{\bbE_{\substack{#1}}} \newcommand{\vbig}{\vphantom{\bigoplus}} \newcommand{\seteq}{\mathrel{\mathop:}=} \newcommand{\defeq}{\stackrel{\mathrm{def}}=} \newcommand{\Mid}{\mathrel{}\middle|\mathrel{}} \newcommand{\Ind}{\mathbf 1} \newcommand{\bits}{\{0,1\}} \newcommand{\sbits}{\{\pm 1\}} \newcommand{\R}{\mathbb R} \newcommand{\Rnn}{\R_{\ge 0}} \newcommand{\N}{\mathbb N} \newcommand{\Z}{\mathbb Z} \newcommand{\Q}{\mathbb Q} \newcommand{\mper}{\,.} \newcommand{\mcom}{\,,} \DeclareMathOperator{\Id}{Id} \DeclareMathOperator{\cone}{cone} \DeclareMathOperator{\vol}{vol} \DeclareMathOperator{\val}{val} \DeclareMathOperator{\opt}{opt} \DeclareMathOperator{\Opt}{Opt} \DeclareMathOperator{\Val}{Val} \DeclareMathOperator{\LP}{LP} \DeclareMathOperator{\SDP}{SDP} \DeclareMathOperator{\Tr}{Tr} \DeclareMathOperator{\Inf}{Inf} \DeclareMathOperator{\poly}{poly} \DeclareMathOperator{\polylog}{polylog} \DeclareMathOperator{\argmax}{arg\,max} \DeclareMathOperator{\argmin}{arg\,min} \DeclareMathOperator{\qpoly}{qpoly} \DeclareMathOperator{\qqpoly}{qqpoly} \DeclareMathOperator{\conv}{conv} \DeclareMathOperator{\Conv}{Conv} \DeclareMathOperator{\supp}{supp} \DeclareMathOperator{\sign}{sign} \DeclareMathOperator{\mspan}{span} \DeclareMathOperator{\mrank}{rank} \DeclareMathOperator{\E}{\mathbb E} \DeclareMathOperator{\pE}{\tilde{\mathbb E}} \DeclareMathOperator{\Pr}{\mathbb P} \DeclareMathOperator{\Span}{Span} \DeclareMathOperator{\Cone}{Cone} \DeclareMathOperator{\junta}{junta} \DeclareMathOperator{\NSS}{NSS} \DeclareMathOperator{\SA}{SA} \DeclareMathOperator{\SOS}{SOS} \newcommand{\iprod}[1]{\langle #1 \rangle} \newcommand{\R}{\mathbb{R}} \newcommand{\cE}{\mathcal{E}} \newcommand{\E}{\mathbb{E}} \newcommand{\pE}{\tilde{\mathbb{E}}} \newcommand{\N}{\mathbb{N}} \renewcommand{\P}{\mathcal{P}} \notag \]
\[ \newcommand{\sleq}{\ensuremath{\preceq}} \newcommand{\sgeq}{\ensuremath{\succeq}} \newcommand{\diag}{\ensuremath{\mathrm{diag}}} \newcommand{\support}{\ensuremath{\mathrm{support}}} \newcommand{\zo}{\ensuremath{\{0,1\}}} \newcommand{\pmo}{\ensuremath{\{\pm 1\}}} \newcommand{\uppersos}{\ensuremath{\overline{\mathrm{sos}}}} \newcommand{\lambdamax}{\ensuremath{\lambda_{\mathrm{max}}}} \newcommand{\rank}{\ensuremath{\mathrm{rank}}} \newcommand{\Mslow}{\ensuremath{M_{\mathrm{slow}}}} \newcommand{\Mfast}{\ensuremath{M_{\mathrm{fast}}}} \newcommand{\Mdiag}{\ensuremath{M_{\mathrm{diag}}}} \newcommand{\Mcross}{\ensuremath{M_{\mathrm{cross}}}} \newcommand{\eqdef}{\ensuremath{ =^{def}}} \newcommand{\threshold}{\ensuremath{\mathrm{threshold}}} \newcommand{\vbls}{\ensuremath{\mathrm{vbls}}} \newcommand{\cons}{\ensuremath{\mathrm{cons}}} \newcommand{\edges}{\ensuremath{\mathrm{edges}}} \newcommand{\cl}{\ensuremath{\mathrm{cl}}} \newcommand{\xor}{\ensuremath{\oplus}} \newcommand{\1}{\ensuremath{\mathrm{1}}} \notag \]
\[ \newcommand{\transpose}[1]{\ensuremath{#1{}^{\mkern-2mu\intercal}}} \newcommand{\dyad}[1]{\ensuremath{#1#1{}^{\mkern-2mu\intercal}}} \newcommand{\nchoose}[1]{\ensuremath{{n \choose #1}}} \newcommand{\generated}[1]{\ensuremath{\langle #1 \rangle}} \notag \]

Indirection and universality

  • See the NAND<< programming language.
  • Understand how NAND<< can be implemented as syntactic sugar on top of NAND++
  • Understand the construction of a universal NAND<< (and hence NAND++) program.

“All problems in computer science can be solved by another level of indirection”, attributed to David Wheeler.

“The programmer is in the unique position that … he has to be able to think in terms of conceptual hierarchies that are much deeper than a single mind ever needed to face before.”, Edsger Dijkstra, “On the cruelty of really teaching computing science”, 1988.

One of the most significant results we showed for NAND programs is the notion of universality: that a NAND program can evaluate other NAND programs. However, there was a significant caveat in this notion. To evaluate a NAND program of \(s\) lines, we needed to use a bigger number of lines than \(s\). It turns out that NAND++ allows us to “break out of this cycle” and obtain a truly universal NAND++ program \(U\) that can evaluate all other programs, including programs that have more lines than \(U\) itself. (As we’ll see in the next lecture, this is not something special to NAND++ but is a feature of many other computational models.) The existence of such a universal program has far reaching applications, and we will explore them in the rest of this course.

To describe the universal program, it will be convenient for us to introduce some extra “syntactic sugar” for NAND++. We’ll use the name NAND<< for the language of NAND++ with this extra syntactic sugar. The classes of functions computable by NAND++ and NAND<< programs are identical, but NAND<< can sometimes be more convenient to work with. Moreover, NAND<< will be useful for us later in the course when we will turn to modelling running time of algorithms.Looking ahead, as we will see in the next lecture, NAND++ programs are essentially equivalent to Turing Machines (more precisely, their single-tape, oblivious variant), while NAND<< programs are equivalent to RAM machines. Turing machines are typically the standard model used in computability and complexity theory, while RAM machines are used in algorithm design. As we will see, their powers are equivalent up to polynomial factors in the running time.

The NAND<< programming language

We now define a seemingly more powerful programming language than NAND++: NAND<< (pronounced “NAND shift”). NAND<< has some additional operators, but as we will see, it can ultimately be implemented by applying certain “syntactic sugar” constructs on top of NAND++. Nonetheless, NAND<< will still serve (especially later in the course) as a useful computational model.If you have encountered computability or computational complexity before, we can already “let you in on the secret”. NAND++ is equivalent to the model known as single tape oblivious Turing machines, while NAND<< is (essentially) equivalent to the model known as RAM machines. For the purposes of the current lecture, the NAND++/Turing-Machine model is indistinguishable from the NAND<</RAM-Machine model (due to a notion known as “Turing completeness”) but the difference between them can matter if one is interested in a fine enough resolution of computational efficiency. There are two key differences between NAND<< and NAND:

  1. The NAND<< programming language works with integer valued as opposed to binary variables.
  2. NAND<< allows indirection in the sense of accessing the bar-th location of an array foo. Specifically, since we use integer valued variables, we can assign the value of bar to the special index i and then use foo_i.

We will allow the following operations on variables:Below foo, bar and baz are indexed or non-indexed variable identifiers (e.g., they can have the form blah or blah_12 or blah_i), as usual, we identify an indexed identifier blah with blah_0. Except for the assignment, where i can be on the lefthand side, the special index variable i cannot be involved in these operations.

  • foo := bar or i := bar (assignment)
  • foo := bar + baz (addition)
  • foo := bar - baz (subtraction)
  • foo := bar >> baz (right shift: \(foo \leftarrow \floor{bar \cdots 2^{-baz}}\))
  • foo := bar << baz (left shift: \(foo \leftarrow bar \cdots 2^{baz}\))
  • foo := bar % baz (modular reduction)
  • foo := bar * baz (multiplication)
  • foo := bar / baz (integer division: \(foo \leftarrow \floor{\tfrac{bar}{baz}}\))
  • foo := bar bAND baz (bitwise AND)
  • foo := bar bXOR baz (bitwise XOR)
  • foo := bar > baz (greater than)
  • foo := bar < baz (smaller than)
  • foo := bar == baz (equality)

The semantics of these operations are as expected except that we maintain the invariant that all variables always take values between \(0\) and the current value of the iteration counter (i.e., number of iterations of the program that have been completed). If an operation would result in assigning to a variable foo a number that is smaller than \(0\), then we assign \(0\) to foo, and if it assigns to foo a number that is larger than the iteration counter, then we assign the value of the iteration counter to foo. Just like C, we interpret any nonzero value as “true” or \(1\), and hence foo := bar NAND baz will assign to foo the value \(0\) if both bar and baz are not zero, and \(1\) otherwise.

Apart from those operations, NAND<< is identical to NAND++. For consistency, we still treat the variable i as special, in the sense that we only allow it to be used as an index, even though the other variables contain integers as well, and so we don’t allow variables such as foo_bar though we can simulate it by first writing i := bar and then foo_i. We also maintain the invariant that at the beginning of each iteration, the value of i is set to the same value that it would have in a NAND++ program (i.e., the function of the iteration counter stated in Reference:computeidx-ex), though this can be of course overwritten by explicitly assigning a value to i. Once again, see the appendix for a more formal specification of NAND<<.

Most of the time we will be interested in applying NAND<< programs on bits, and hence we will assume that both inputs and outputs are bits. We can enforce the latter condition by not allowing y_ variables to be on the lefthand side of any operation other than NAND. However, the same model can be used to talk about functions that map tuples of integers to tuples of integers, and so we may very occasionally abuse notation and talk about NAND<< programs that compute on integers.

Simulating NAND<< in NAND++

The most important fact we need to know about NAND<< is that it can be implemented by mere “syntactic sugar” and hence does not give us more computational power than NAND++, as stated in the following theorem:

For every (partial) function \(F:\{0,1\}^* \rightarrow \{0,1\}^*\), \(F\) is computable by a NAND++ program if and only if \(F\) is computable by a NAND<< program.

The rest of this section is devoted to outlining the proof of Reference:NANDequiv-thm. The “only if” direction of the theorem is immediate. After all, every NAND++ program \(P\) is in particular also a NAND<< program, and hence if \(F\) is computable by a NAND++ program then it is also computable by a NAND<< program. To show the “if” direction, we need to show how we can implement all the operations of NAND<< in NAND++. In other words, we need to give a “NAND<< to NAND++ compiler”.

Writing a compiler in full detail, and then proving that it is correct, is possible (and has been done) but is quite a time consuming enterprise, and not very illuminating. For our purposes, we need to convince ourselves that Reference:NANDequiv-thm and that such a transformation exists, and we will do so by outlining the key ideas behind it.The webpage nandpl.org should eventually contain a program that transforms a NAND<< program into an equivalent NAND++ program.

Let \(P\) be a NAND<< program, we need to transform \(P\) into a NAND++ program \(P'\) that computes the same function as \(P\). The idea will be that \(P'\) will simulate \(P\) “in its belly”. We will use the variables of \(P'\) to encode the state of the simulated program \(P\), and every single NAND<< step of the program \(P\) will be translated into several NAND++ steps by \(P'\). We will do so in several steps:

Step 1: Controlling the index and inner loops. We have seen that we can add syntactic sugar for inner loops and incrementing/derementing the index (i.e., operations such as i++ (foo) and i-- (bar)) to NAND++. Hence we can assume access to these operations in constructing \(P'\). In particular, we can use such an inner loop to perform tasks such as copying the contents of one array (e.g., variables foo_0, \(\ldots\), foo_\(\expr{k-1}\) for some \(k\)) to another.

Step 2: Operations on integers. We can use some standard prefix free encoding to represent an integer as an array of bits. For example, we can use the map \(pf:\Z \rightarrow \{0,1\}^*\) defined as follows. Given an integer \(z\in \Z\), if \(z \geq 0\) then we define the string \(pf(z)\) as \(z_0z_0z_1z_1 \ldots z_{n-1}z_{n-1}01\) (where \(n\) is the smallest number s.t. \(2^n > z\) and \(z_i\) is the binary digit of \(x\) corresponding to \(2^i\)). If \(z<0\) then we define \(pf(z)=10pf(|z|)\). We can then use the standard gradeschool algorithms to define NAND++ macros that perform arithmetic operations on the representation of integers (e.g., addition, multiplication, division, etc.).

Step 3: Move index to specified location. We can use the above operations to move the index to a location encoded by an integer. To do so, we can first move i to the zero location by decreasing it until the atstart_i equals \(1\) (where, as we’ve seen before, atstart is an array we can set up so attstart_0 is \(1\) and atstart_\(\expr{j}\) is zero for \(j\neq 0\)). Now, if the variables foo_0,foo_1,… encode the number \(k\in \N\) we can set the value of i to \(k\) as follows. We’ll set bar to \(1\) and an inner loop that will proceed as long as bar is not zero. In this loop we will do the following: (1) If foo encodes \(0\) then set bar to zero. (2) Otherwise, we use a nested inner loop to decrement the number represented by foo by \(1\), and perform the operation i++ (bar).

Step 4: Maintaining an iteration counter and index. The NAND++ program \(P'\) will simulate execution of the NAND<< program \(P\). Every step of \(P\) will be simulated by several steps of \(P'\). We can use the above operations to maintain a variable itercounter and index that will encode the current step of \(P\) that is being executed and the current value of the special index variable i in the simulated program \(P\) (which does not have to be the same as the value of i in the NAND++ program \(P'\)).

Step 5: Embedding two dimensional arrays into one dimension. If foo and bar the encode the natural numbers \(x,y \in \N\), then we can use NAND++ to compute the map \(PAIR:\N^2 \rightarrow \N\) where \(PAIR(x,y) = \tfrac{1}{2}(x+y)(x+y+1)+x\). In Reference:pair-ex we ask you to verify that \(PAIR\) is a one-to-one map from \(\N^2\) to \(\N\) and that there are NAND++ programs \(P_0,P_1\) such that for every \(x_0,x_1 \in \N\) and \(i \in \{0,1\}\), \(P_i(PAIR(x_0,x_1))=x_i\). Using this \(PAIR\) map, we can assume we have access to two dimensional arrays in our NAND++ program.

Step 6: Embedding an array of integers into a two dimensional bit array. We can use the same encoding as above to embed a one-dimensional array foo of integers into a two-dimensional array bar of bits, where bar_{\(\expr{i}\), \(\expr{j}\)} will encode the \(j\)-th bit in the representation of the integer foo_\(\expr{i}\). Thus we can simulate the integer arrays of the NAND<< program \(P\) in the NAND++ program \(P'\).

Step 7: Simulating \(P\). Now we have all the components in place to simulate every operation of \(P\) in \(P'\). The program \(P'\) will have a two dimensional bit array corresponding to any one dimensional array of \(P\), as well as variables to store the iteration counter, index, as well as the loop variable of the simulated program \(P\). Every step of \(P\) can now be translated into an inner loop that would perform the same operation on the representations of the state.

We omit the full details of all the steps above and their analysis, which are tedious but not that insightful.

Example

Here is a program that computes the function \(PALINDROME:\{0,1\}^* \rightarrow \{0,1\}\) that outputs \(1\) on \(x\) if and only if \(x_i = x_{|x|-i}\) for every \(i\in \{0,\ldots, |x|-1\}\). This program uses NAND<< with the syntactic sugar we described before, but as discussed above, we can transform it into a NAND++ program.

// A sample NAND<< program that computes the language of palindromes
// By Juan Esteller
def a := NOT(b) {
  a := b NAND b
}
o := NOT(z)
two := o + o
if(NOT(seen_0)) {
  cur := z
  seen_0 := o
}
i := cur
if(validx_i) {
 cur := cur + o  
 loop := o
}
if(NOT(validx_i)) {
  computedlength := o
}
if(computedlength) {
  if(justentered) {
    justentered := o
    iter := z
  }
  i := iter
  left := x_i
  i := (cur - iter) - o
  right := x_i
  if(NOT(left == right)) {   
    loop := z
    y_0 := z
  }
  halflength := cur / two
  if(NOT(iter < halflength)) {
   y_0 := o
   loop := z
  }  
  iter := iter + o  
}

The “Best of both worlds” paradigm

The equivalence between NAND++ and NAND<< allows us to choose the most convenient language for the task at hand:

  • When we want to give a theorem about all programs, we can use NAND++ because it is simpler and easier to analyze. In particular, if we want to show that a certain function can not be computed, then we will use NAND++.
  • When we want to show the existence of a program computing a certain function, we can use NAND<<, because it is higher level and easier to program in. In particular, if we want to show that a function can be computed then we can use NAND<<. In fact, because NAND<< has much of the features of high level programming languages, we will often describe NAND<< programs in an informal manner, trusting that the reader can fill in the details and translate the high level description to the precise program. (This is just like the way people typically use informal or “pseudocode” descriptions of algorithms, trusting that their audience will know to translate these descriptions to code if needed.)

Our usage of NAND++ and NAND<< is very similar to the way people use in practice high and low level programming languages. When one wants to produce a device that executes programs, it is convenient to do so for very simple and “low level” programming language. When one wants to describe an algorithm, it is convenient to use as high level a formalism as possible.

By having the two equivalent languages NAND++ and NAND<<, we can “have our cake and eat it too”, using NAND++ when we want to prove that programs can’t do something, and using NAND<< or other high level languages when we want to prove that programs can do something.

One high level tool we can use in describing NAND<< programs is recursion. We can use the standard implementation of the stack data structure, which can be (and in fact is) used to implement recursion. A stack is a data structure containing a sequence of elements, where we can “push” elements into it and “pop” them from it in “first in last out” order. We can implement stack by an array of integers stack_0, \(\ldots\), stack_\(\expr{k-1}\) and stackpointer will be the number \(k\) of items in the stack. We implement push(foo) by doing i := stackpointer and stack_i := foo and pop() by letting stackpointer := stackpointer - 1. By encoding strings as integers, we can have allow strings in our stack as well.

Now we can implement recursion using the stack just as is done in most programming languages. The idea is that a (recursive or non recursive) call to a function \(F\) is implemented by pushing the arguments for \(F\) into the stack. The code of \(F\) will “pop” the arguments from the stack, perform the computation (which might involve making recursive or non recursive calls) and then “push” its return value into the stack. Because of the “first in last out” nature of a stack, we do not return control to the calling procedure until all the recursive calls are done.

Specifically, we note that using loops and conditionals, we can implement “goto” statements in NAND<<. Moreover, we can even implement “dynamic gotos”, in the sense that we can set integer labels for certain lines of codes, and have a goto foo operation that moves execution to the line labeled by foo. Now, if we want to make a call to a function \(F\) with parameter bar then we will push into the stack the label of the next line and bar, and then make a goto to the code of \(F\). That code will pop its parameter from the stack, do the computation of \(F\), and when it needs to resume execution, will pop the label from the stack and goto there.

You can find online a tutorial on how recursion is implemented via stack in your favorite programming language, whether it’s Python , JavaScript, or Lisp/Scheme.

Let’s talk about abstractions.

At some point in any theory of computation course, the instructor and students need to have the talk. That is, we need to discuss the level of abstraction in describing algorithms. In algorithms courses, one typically describes algorithms in English, assuming readers can “fill in the details” and would be able to convert such an algorithm into an implementation if needed. For example, we might describe the breadth first search algorithm to find if two vertices \(u,v\) are connected as follows:

  1. Put \(u\) in queue \(Q\).
  2. While \(Q\) is not empty:
  1. Declare “unconnected”.

We call such a description a high level description.

If we wanted to give more details on how to implement breadth first search in a programming language such as Python or C (or NAND<< / NAND++ for that matter), we would describe how we implement the queue data structure using an array, and similarly how we would use arrays to implement the marking. We call such a “intermediate level” description an implementation level or pseudocode description. Finally, if we want to describe the implementation precisely, we would give the full code of the program (or another fully precise representation, such as in the form of a list of tuples). We call this a formal or low level description.

While initially we might have described NAND, NAND++, and NAND<< programs at the full formal level (and the NAND website contains more such examples), as the course continues we will move to implementation and high level description. After all, our focus is typically not to use these models for actual computation, but rather to analyze the general phenomenon of computation. That said, if you don’t understand how the high level description translates to an actual implementation, you should always feel welcome to ask for more details of your teachers and teaching fellows.

A similar distinction applies to the notion of representation of objects as strings. Sometimes, to be precise, we give a low level specification of exactly how an object maps into a binary string. For example, we might describe an encoding of \(n\) vertex graphs as length \(n^2\) binary strings, by saying that we map a graph \(G\) over the vertex \([n]\) to a string \(x\in \{0,1\}^{n^2}\) such that the \(n\cdot i + j\)-th coordinate of \(x\) is \(1\) if and only if the edge \(\overrightarrow{i \; j}\) is present in \(G\). We can also use an intermediate or implementation level description, by simply saying that we represent a graph using the adjacency matrix representation. Finally, because we translating between the various representations of graphs (and objects in general) can be done via a NAND<< (and hence a NAND++) program, when talking in a high level we also suppress discussion of representation altogether. For example, the fact that graph connectivity is a computable function is true regardless of whether we represent graphs as adjacency lists, adjacency matrices, list of edge-pairs, and so on and so forth. Hence, in cases where the precise representation doesn’t make a difference, we would often talk about our algorithms as taking as input an object \(O\) (that can be a graph, a vector, a program, etc.) without specifying how \(O\) is encoded as a string.

Universality: A NAND++ interpreter in NAND++

Like a NAND program, a NAND++ or a NAND<< program is ultimately a sequence of symbols and hence can obviously be represented as a binary string. We will spell out the exact details of representation later, but as usual, the details are not so important (e.g., we can use the ASCII encoding of the source code). What is crucial is that we can use such representation to evaluate any program. That is, we prove the following theorem:

There is a NAND++ program \(U\) that computes the partial function \(EVAL:\{0,1\}^* \rightarrow \{0,1\}^*\) defined as follows: \[ EVAL(P,x)=P(x) \] for strings \(P,x\) such that \(P\) is a valid representation of a NAND++ program which produces an output on \(x\). Moreover, for every input \(x\in \{0,1\}^*\) on which \(P\) does not halt, \(U(P,x)\) does not halt as well.

This is a stronger notion than the universality we proved for NAND, in the sense that we show a single universal NAND++ program \(U\) that can evaluate all NAND programs, including those that have more lines than the lines in \(U\). In particular, \(U\) can even be used to evaluate itself! This notion of self reference will appear time and again in this course, and as we will see, leads to several counterintuitive phenomena in computing.

Because we can easily transform a NAND<< program into a NAND++ program, this means that even the seemingly “weaker” NAND++ programming language is powerful enough to simulate NAND<< programs. Indeed, as we already alluded to before, NAND++ is powerful enough to simulate also all other standard programming languages such as Python, C, Lisp, etc.

Representing NAND++ programs as strings

Before we can prove Reference:univnandppnoneff, we need to make its statement precise by specifying a representation scheme for NAND++ programs. As mentioned above, simply representing the program as a string using ASCII or UTF-8 encoding will work just fine, but we will use a somewhat more convenient and concrete representation, which is the natural generalization of the “list of triples” representation for NAND programs. We will assume that all variables are of the form foo_## where foo is an identifier and ## is some number or the index i. If a variable foo does not have an index then we add the index zero to it. We represent an instruction of the form

foo_\(\expr{j}\) := bar_\(\expr{k}\) NAND baz_\(\expr{\ell}\)

as a \(6\) tuple \((a,j,b,k,c,\ell)\) where \(a,b,c\) are numbers corresponding to the labels foo,bar,and baz respectively, and \(j,k,\ell\) are the corresponding indices. For variables that indexed by the special index i, we will encode the index by \(s\), where \(s\) ithe number of lines in the program. (There is no risk of conflict since we did not allow numerical indices larger or equal to the number of lines in the program.) We will set the identifiers of x,y,validx and loop to \(0,1,2,3\) respectively. Therefore the representation of the parity program

tmp_1 := seen_i NAND seen_i
tmp_2 := x_i NAND tmp_1
val := tmp_2 NAND tmp_2
ns := s NAND s
y_0 := ns NAND ns
u := val NAND s
v := s NAND u
w := val NAND u
s := v NAND w
seen_i := z NAND z
stop := validx_i NAND validx_i
loop := stop NAND stop

will be

[[4, 1, 5, 11, 5, 11],
 [4, 2, 0, 11, 4, 1],
 [6, 0, 4, 2, 4, 2],
 [7, 0, 8, 0, 8, 0],
 [1, 0, 7, 0, 7, 0],
 [9, 0, 6, 0, 8, 0],
 [10, 0, 8, 0, 9, 0],
 [11, 0, 6, 0, 9, 0],
 [8, 0, 10, 0, 11, 0],
 [5, 11, 12, 0, 12, 0],
 [13, 0, 2, 11, 2, 61],
 [3, 0, 13, 0, 13, 0]]

Binary encoding: The above is a way to represent any NAND++ program as a list of numbers. We can of course encode such a list as a binary string in a number of ways. For concreteness, since all the numbers involved are between \(0\) and \(s\) (where \(s\) is the number of lines), we can simply use a string of length \(6\ceil{\log (s+1)}\) to represent them, starting with the prefix \(0^{s+1}1\) to encode \(s\). For convenience we will assume that any string that is not formatted in this way encodes the single line program y_0 := x_0 NAND x_0. This way we can assume that every string \(P\in\bits^*\) represents some NAND++ program.

A NAND++ interpreter in NAND<<

Here is the “pseudocode”/“sugar added” version of an interpreter for NAND++ programs (given in the list of 6 tuples representation) in NAND<<. We assume below that the input is given as integers x_0,\ldots,x_\(\expr{6\cdot lines-1}\) where \(lines\) is the number of lines in the program. We also assume that NumberVariables gives some upper bound on the total number of distinct non-indexed identifiers used in the program (we can also simply use \(lines\) as this bound).

simloop := 3
totalvars := NumberVariables(x)
maxlines  := Length(x) / 6
currenti := 0
currentround := 0
increasing := 1
pc := 0
while (true) {
    line := 0
    foo    :=  x_{6*line + 0}
    fooidx :=  x_{6*line + 1}
    bar    :=  x_{6*line + 2}
    baridx :=  x_{6*line + 3}
    baz    :=  x_{6*line + 4}
    bazidx :=  x_{6*line + 5}
    if (fooidx == maxlines) {
        fooidx := currenti
    }
    ... // similar for baridx, bazidx

    vars_{totalvars*fooidx+foo} := vars_{totalvars*baridx+bar} NAND vars_{totalvars*bazidx+baz}
    line++

    if line==maxlines {
        if not avars[simloop] {
            break
        }
        pc := pc+1
        if (increasing) {
            i := i + 1
        } else
        {
            i := i - 1
        }
        if i>r {
            increasing := 0
            r := r+1
        }
        if i==0 {
            increasing := 1
        }

    }
    // keep track in loop above of largest m that y_{m-1} was assigned a value
    // add code to move vars[0*totalvars+1]...vars[(m-1)*totalvars+1] to y_0..y_{m-1}
}

Since we can transform every NAND<< program to a NAND++ one, we can also implement this interpreter in NAND++, hence completing the proof of Reference:univnandppnoneff.

A Python interpreter in NAND++

At this point you probably can guess that it is possible to write an interpreter for languages such as C or Python in NAND<< and hence in NAND++ as well. After all, with NAND++ / NAND<< we have access to an unbounded array of memory, which we can use to simulate memory allocation and access, and can do all the basic computation steps offered by modern CPUs. Writing such an interpreter is nobody’s idea of a fun afternoon, but the fact it can be done gives credence to the belief that NAND++ is a good model for general-purpose computing.

Lecture summary

Exercises

Let \(PAIR:\N^2 \rightarrow \N\) be the function defined as \(PAIR(x_0,x_1)= \tfrac{1}{2}(x_0+x_1)(x_0+x_1+1) + x_1\).
1. Prove that for every \(x^0,x^1 \in \N\), \(PAIR(x^0,x^1)\) is indeed a natural number.
2. Prove that \(PAIR\) is one-to-one
3. Construct a NAND++ program \(P\) such that for every \(x^0,x^1 \in \N\), \(P(pf(x^0)pf(x^1))=pf(PAIR(x^0,x^1))\), where \(pf\) is the prefix-free encoding map defined above. You can use the syntactic sugar for inner loops, conditionals, and incrementing/decrementing the counter.
4. Construct NAND++ programs \(P_0,P_1\) such that for for every \(x^0,x^1 \in \N\) and \(i \in N\), \(P_i(pf(PAIR(x^0,x^1)))=pf(x^i)\). You can use the syntactic sugar for inner loops, conditionals, and incrementing/decrementing the counter.

Prove that for every \(F:\{0,1\}^* \rightarrow \{0,1\}^*\), the function \(F\) is computable if and only if the following function \(G:\{0,1\}^* \rightarrow \{0,1\}\) is computable, where \(G\) is defined as follows: \(G(x,i,\sigma) = \begin{cases} F(x)_i & i < |F(x)|, \sigma =0 \\ 1 & i < |F(x)|, \sigma = 1 \\ 0 & i \geq |F(x)| \end{cases}\)

Bibliographical notes

The notion of “NAND++ programs” we use is nonstandard but (as we will see) they are equivalent to standard models used in the literature. Specifically, NAND++ programs are closely related (though not identical) to oblivious one-tape Turing machines, while NAND<< programs are essentially the same as RAM machines. As we’ve seen in these lectures, in a qualitative sense these two models are also equivalent to one another, though the distinctions between them matter if one cares (as is typically the case in algorithms research) about polynomial factors in the running time.

Further explorations

Some topics related to this lecture that might be accessible to advanced students include: (to be completed)

Acknowledgements