Add MacKay exercise 8.11, part 1: the forward direction

_subfiles/_exr-forward-n-gram-conditional-entropy.qmd

@@ -0,0 +1,14 @@
+::: {#exr-forward-n-gram-conditional-entropy}
+#### Conditional entropy of an $N$-gram model read forward
+
+Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence)
+that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$.
+For every $t > N$,
+show that the forward [conditional entropy](#def-conditional-entropy) of $X_t$ is
+
+$$
+\entropyf{X_t \mid X_1, \ldots, X_{t-1}}
+= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}.
+$$
+
+:::

_subfiles/_exr-reversed-text-total-entropy.qmd

@@ -0,0 +1,17 @@
+::: {#exr-reversed-text-total-entropy}
+#### The whole text costs the same in both orders
+
+Let $X_1, \ldots, X_n$ be [discrete random variables](https://morrison-lab.github.io/pds/random-variables.html#def-discrete-rv).
+Use the [chain rule for entropy](#thm-entropy-chain-rule-n),
+with the variables listed in each of two orders,
+to show that
+
+$$
+\sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}}
+= \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n},
+$$
+
+where the term for $t = 1$ on the left and the term for $t = n$ on the right
+are unconditional [entropies](https://morrison-lab.github.io/pds/expectation.html#def-entropy).
+
+:::

_subfiles/_rem-reversed-text-guessing.qmd

@@ -0,0 +1,13 @@
+:::{#rem-reversed-text-guessing}
+#### Guessing a text backward
+
+A reader can guess a text one character at a time from its start,
+guessing each character from the characters already read.
+The reader can also guess it from its end,
+guessing each character from the characters that follow it.
+Most people find the backward game harder.
+That raises a question:
+does a reversed text carry more information per character than the text read forward
+[@mackay2003information, exercise 8.11]?
+
+:::

_subfiles/_sol-forward-n-gram-conditional-entropy.qmd

@@ -0,0 +1,66 @@
+::: {#sol-forward-n-gram-conditional-entropy}
+
+To shorten the displays,
+write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$,
+and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$.
+Each value's subscript says which variable takes it.
+
+For values $x_1, \ldots, x_{t-1}$ with $p(x_1, \ldots, x_{t-1}) > 0$,
+@def-entropy-given-value makes $\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}}$
+a [function](https://morrison-lab.github.io/mcs/functions.html#def-function) of the [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p(\cdot \mid x_1, \ldots, x_{t-1})$ alone.
+The [$N$-gram model](#def-n-gram-model) makes that conditional probability mass function equal to $p(\cdot \mid x_{t-N+1}, \ldots, x_{t-1})$,
+so
+
+$$
+\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}}
+= \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}}.
+$$
+
+Then
+
+$$
+\begin{aligned}
+\entropyf{X_t \mid X_1, \ldots, X_{t-1}}
+&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} \\
+&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\
+&= \sum_{x_{t-N+1}, \ldots, x_{t-1}} p(x_{t-N+1}, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\
+&= \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}.
+\end{aligned}
+$$
+
+The first line is @thm-conditional-entropy-average,
+with $(X_1, \ldots, X_{t-1})$ as the conditioning variable.
+The second line uses the display above.
+The third line sums over $x_1, \ldots, x_{t-N}$ first:
+the second factor of each term does not depend on them,
+and summing $p(x_1, \ldots, x_{t-1})$ over them gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t-N+1}, \ldots, x_{t-1})$.
+The fourth line is @thm-conditional-entropy-average again.
+
+The two-variable [chain rule](#thm-entropy-chain-rule),
+with $(X_{t-N+1}, \ldots, X_{t-1})$ as its first variable and $X_t$ as its second,
+gives
+
+$$
+\entropyf{X_{t-N+1}, \ldots, X_t}
+= \entropyf{X_{t-N+1}, \ldots, X_{t-1}} + \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}.
+$$
+
+Subtracting $\entropyf{X_{t-N+1}, \ldots, X_{t-1}}$ from both sides gives
+
+$$
+\entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}
+= \entropyf{X_{t-N+1}, \ldots, X_t} - \entropyf{X_{t-N+1}, \ldots, X_{t-1}}.
+$$
+
+An [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) depends only on the [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf).
+The [stationary sequence](#def-stationary-sequence) with shift $s = t - N$ gives
+$(X_{t-N+1}, \ldots, X_t)$ the same joint probability mass function as $(X_1, \ldots, X_N)$,
+and $(X_{t-N+1}, \ldots, X_{t-1})$ the same one as $(X_1, \ldots, X_{N-1})$.
+So
+
+$$
+\entropyf{X_t \mid X_1, \ldots, X_{t-1}}
+= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}.
+$$
+
+:::

_subfiles/_sol-reversed-text-total-entropy.qmd

@@ -0,0 +1,34 @@
+::: {#sol-reversed-text-total-entropy}
+
+Listing the same variables in another order does not change their [joint entropy](#def-joint-entropy),
+because the reordered vector takes each reordered value with the same [probability](https://morrison-lab.github.io/pds/probability-basics.html#def-probability),
+as in the proof of @cor-entropy-chain-rule-reversed.
+For the same reason,
+reordering the variables in a condition does not change a [conditional entropy](#def-conditional-entropy).
+The [chain rule](#thm-entropy-chain-rule-n), with the variables in their usual order, gives
+
+$$\entropyf{X_1, \ldots, X_n} = \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}}.$$
+
+With the variables listed as $X_n, X_{n-1}, \ldots, X_1$,
+the $i$th variable is $X_{n+1-i}$,
+and the variables before it are $X_n, \ldots, X_{n+2-i}$.
+So
+
+$$
+\begin{aligned}
+\entropyf{X_1, \ldots, X_n}
+&= \entropyf{X_n, \ldots, X_1} \\
+&= \sum_{i=1}^{n} \entropyf{X_{n+1-i} \mid X_n, \ldots, X_{n+2-i}} \\
+&= \sum_{t=1}^{n} \entropyf{X_t \mid X_n, \ldots, X_{t+1}} \\
+&= \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}.
+\end{aligned}
+$$
+
+The first line reorders the joint entropy,
+the second is the chain rule,
+the third substitutes $t = n + 1 - i$,
+and the fourth reorders each condition.
+Both sums equal $\entropyf{X_1, \ldots, X_n}$,
+so they equal each other.
+
+:::

entropy.qmd

@@ -398,6 +398,26 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*.
 
 {{< include ids/_subfiles/_exm-not-n-gram-model.qmd >}}
 
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_rem-reversed-text-guessing.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_exr-reversed-text-total-entropy.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_sol-reversed-text-total-entropy.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_exr-forward-n-gram-conditional-entropy.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include ids/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd >}}
+
 ## References {.unnumbered}
 
 ::: {#refs}