Add the backward half of MacKay exercise 8.11

_subfiles/_exr-backward-n-gram-conditional-entropy.qmd

@@ -0,0 +1,15 @@
+::: {#exr-backward-n-gram-conditional-entropy}
+#### Conditional entropy of an $N$-gram model read backward
+
+Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence)
+that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$,
+let $n \ge N$,
+and let $1 \le t \le n - N + 1$.
+Use @exr-reversed-n-gram-model to show that the backward [conditional entropy](#def-conditional-entropy) of $X_t$ is
+
+$$
+\entropyf{X_t \mid X_{t+1}, \ldots, X_n}
+= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}.
+$$
+
+:::

_subfiles/_exr-reversed-n-gram-model.qmd

@@ -0,0 +1,16 @@
+::: {#exr-reversed-n-gram-model}
+#### An $N$-gram model read backward
+
+Let $X_1, X_2, \ldots$ be an [$N$-gram model](#def-n-gram-model) with $N \ge 2$,
+let $n \ge N$,
+and let $1 \le t \le n - N + 1$.
+Show that given all the later variables,
+$X_t$ depends only on the $N - 1$ variables just after it:
+for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$,
+
+$$
+\P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n)
+= \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}).
+$$
+
+:::

_subfiles/_exr-reversed-text-information.qmd

@@ -0,0 +1,8 @@
+::: {#exr-reversed-text-information}
+#### Which guessing game is harder?
+
+Use @exr-reversed-text-total-entropy, @exr-forward-n-gram-conditional-entropy and @exr-backward-n-gram-conditional-entropy
+to compare the information per character in a reversed text and in the text read forward,
+and use that comparison to explain the difference in difficulty described in @rem-reversed-text-guessing.
+
+:::

_subfiles/_sol-backward-n-gram-conditional-entropy.qmd

@@ -0,0 +1,44 @@
+::: {#sol-backward-n-gram-conditional-entropy}
+
+To shorten the displays,
+write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$.
+Each value's subscript says which variable takes it.
+
+The reasoning of @sol-forward-n-gram-conditional-entropy carries over,
+with the later variables in place of the earlier ones
+and @exr-reversed-n-gram-model in place of the [$N$-gram model](#def-n-gram-model):
+
+$$
+\begin{aligned}
+\entropyf{X_t \mid X_{t+1}, \ldots, X_n}
+&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\
+&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\
+&= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\
+&= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}.
+\end{aligned}
+$$
+
+The first and fourth lines are @thm-conditional-entropy-average.
+The second line uses @exr-reversed-n-gram-model and @def-entropy-given-value.
+The third line sums over $x_{t+N}, \ldots, x_n$ first,
+which gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t+1}, \ldots, x_{t+N-1})$.
+
+The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable,
+then gives
+
+$$
+\entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}
+= \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}.
+$$
+
+The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives
+$(X_t, \ldots, X_{t+N-1})$ the same [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf) as $(X_1, \ldots, X_N)$,
+and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$.
+So
+
+$$
+\entropyf{X_t \mid X_{t+1}, \ldots, X_n}
+= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}.
+$$
+
+:::

_subfiles/_sol-reversed-n-gram-model.qmd

@@ -0,0 +1,79 @@
+::: {#sol-reversed-n-gram-model}
+
+To shorten the displays,
+write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$,
+and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$.
+Each value's subscript says which variable takes it.
+
+First, for each $s$ with $t + N \le s \le n$,
+the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$:
+for values with $p(x_t, \ldots, x_{s-1}) > 0$,
+
+$$
+\begin{aligned}
+p(x_s \mid x_t, \ldots, x_{s-1})
+&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\
+&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\
+&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\
+&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}).
+\end{aligned}
+$$
+
+The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$.
+The second line is the [$N$-gram model](#def-n-gram-model),
+which applies
+because $s > N$.
+The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum.
+The fourth line uses that a [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) sums to $1$.
+Because $s - N + 1 \ge t + 1$,
+the same holds with $t + 1$ in place of $t$.
+
+Write $R$ for the product of these factors:
+
+$$R = \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}).$$
+
+The product is empty, and $R$ is $1$,
+when $n = t + N - 1$.
+No factor involves $x_t$,
+because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$.
+Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$,
+and then the cut above to each factor,
+gives
+
+$$
+\begin{aligned}
+p(x_t, \ldots, x_n)
+&= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\
+&= p(x_t, \ldots, x_{t+N-1}) \, R.
+\end{aligned}
+$$
+
+The same two steps, starting from $x_{t+1}$, give
+
+$$
+\begin{aligned}
+p(x_{t+1}, \ldots, x_n)
+&= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\
+&= p(x_{t+1}, \ldots, x_{t+N-1}) \, R.
+\end{aligned}
+$$
+
+The left side of the second display is positive by assumption,
+so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$.
+By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob),
+
+$$
+\begin{aligned}
+p(x_t \mid x_{t+1}, \ldots, x_n)
+&= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\
+&= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\
+&= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\
+&= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}).
+\end{aligned}
+$$
+
+The second line substitutes the two displays above,
+the third cancels $R$,
+and the fourth is the definition of conditional probability again.
+
+:::

_subfiles/_sol-reversed-text-information.qmd

@@ -0,0 +1,23 @@
+::: {#sol-reversed-text-information}
+
+@exr-forward-n-gram-conditional-entropy and @exr-backward-n-gram-conditional-entropy give each character away from the ends of the text
+the same [conditional entropy](#def-conditional-entropy) forward and backward,
+and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward.
+So the reversed text carries no more information per character than the text read forward.
+The two games tie only on average:
+in English, "q" is almost always followed by "u",
+but "u" is rarely preceded by "q",
+so a single guess can be easy one way and hard the other.
+
+People find the backward game harder
+because their guesses are worse,
+not because the text holds more information.
+A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $g$
+in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $f$
+spends $\crossentropyf{f}{g}$ [bits](#def-bit) per character on average,
+which exceeds $\entropyf{f}$ by $\kldivf{f}{g}$,
+by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition).
+Years of reading forward make $g$ close to $f$ for forward guessing,
+and no such practice makes it close backward.
+
+:::

entropy.qmd

@@ -418,6 +418,30 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*.
 
 {{< include ids/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd >}}
 
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_exr-reversed-n-gram-model.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_sol-reversed-n-gram-model.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_exr-reversed-text-information.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_sol-reversed-text-information.qmd >}}
+
 ## References {.unnumbered}
 
 ::: {#refs}