From 13de260cfa06c99ef40a8d2e4106d2bf6a832461 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 09:08:38 +0000 Subject: [PATCH 1/3] Add the backward half of MacKay exercise 8.11 Two exercises: a stationary N-gram model read backward is again an N-gram model, and so its backward conditional entropy equals the forward one. The second solution closes the 8.11 question: the reversed text carries no more information per character, and the backward guessing game is harder only because people's backward guesses are worse. Stacked on the PR with the remark and the first two exercises. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- ...xr-backward-n-gram-conditional-entropy.qmd | 18 +++++ _subfiles/_exr-reversed-n-gram-model.qmd | 16 ++++ ...ol-backward-n-gram-conditional-entropy.qmd | 64 +++++++++++++++ _subfiles/_sol-reversed-n-gram-model.qmd | 79 +++++++++++++++++++ entropy.qmd | 16 ++++ 5 files changed, 193 insertions(+) create mode 100644 _subfiles/_exr-backward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_exr-reversed-n-gram-model.qmd create mode 100644 _subfiles/_sol-backward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_sol-reversed-n-gram-model.qmd diff --git a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..597e1bb5 --- /dev/null +++ b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd @@ -0,0 +1,18 @@ +::: {#exr-backward-n-gram-conditional-entropy} +#### Conditional entropy of an $N$-gram model read backward + +Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence) +that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, +let $n \ge N$, +and let $1 \le t \le n - N + 1$. +Use @exr-reversed-n-gram-model to show that the backward [conditional entropy](#def-conditional-entropy) of $X_t$ is + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +Together with @exr-reversed-text-total-entropy and @exr-forward-n-gram-conditional-entropy, +what does this say about the two guessing games in @rem-reversed-text-guessing? + +::: diff --git a/_subfiles/_exr-reversed-n-gram-model.qmd b/_subfiles/_exr-reversed-n-gram-model.qmd new file mode 100644 index 00000000..adbf73ec --- /dev/null +++ b/_subfiles/_exr-reversed-n-gram-model.qmd @@ -0,0 +1,16 @@ +::: {#exr-reversed-n-gram-model} +#### An $N$-gram model read backward + +Let $X_1, X_2, \ldots$ be an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, +let $n \ge N$, +and let $1 \le t \le n - N + 1$. +Show that given all the later variables, +$X_t$ depends only on the $N - 1$ variables just after it: +for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$, + +$$ +\P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n) += \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}). +$$ + +::: diff --git a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..0a47d0fc --- /dev/null +++ b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd @@ -0,0 +1,64 @@ +::: {#sol-backward-n-gram-conditional-entropy} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$. +Each value's subscript says which variable takes it. + +The reasoning of @sol-forward-n-gram-conditional-entropy carries over, +with the later variables in place of the earlier ones +and @exr-reversed-n-gram-model in place of the [$N$-gram model](#def-n-gram-model): + +$$ +\begin{aligned} +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} +&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\ +&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ +&= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ +&= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}. +\end{aligned} +$$ + +The first and fourth lines are @thm-conditional-entropy-average. +The second line uses @exr-reversed-n-gram-model and @def-entropy-given-value. +The third line sums over $x_{t+N}, \ldots, x_n$ first, +which gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t+1}, \ldots, x_{t+N-1})$. + +The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable, +then gives + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}} += \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}. +$$ + +The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives +$(X_t, \ldots, X_{t+N-1})$ the same [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf) as $(X_1, \ldots, X_N)$, +and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$. +So + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +This exercise and @exr-forward-n-gram-conditional-entropy give each character away from the ends of the text +the same [conditional entropy](#def-conditional-entropy) forward and backward, +and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward. +So the reversed text carries no more information per character than the text read forward. +The two games tie only on average: +in English, "q" is almost always followed by "u", +but "u" is rarely preceded by "q", +so a single guess can be easy one way and hard the other. + +People find the backward game harder +because their guesses are worse, +not because the text holds more information. +A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ +in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p$ +spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, +which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, +by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). +Years of reading forward make $q$ close to $p$ for forward guessing, +and no such practice makes it close backward. + +::: diff --git a/_subfiles/_sol-reversed-n-gram-model.qmd b/_subfiles/_sol-reversed-n-gram-model.qmd new file mode 100644 index 00000000..a7c9775a --- /dev/null +++ b/_subfiles/_sol-reversed-n-gram-model.qmd @@ -0,0 +1,79 @@ +::: {#sol-reversed-n-gram-model} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, +and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. +Each value's subscript says which variable takes it. + +First, for each $s$ with $t + N \le s \le n$, +the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$: +for values with $p(x_t, \ldots, x_{s-1}) > 0$, + +$$ +\begin{aligned} +p(x_s \mid x_t, \ldots, x_{s-1}) +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\ +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\ +&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\ +&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}). +\end{aligned} +$$ + +The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. +The second line is the [$N$-gram model](#def-n-gram-model), +which applies +because $s > N$. +The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. +The fourth line uses that a [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) sums to $1$. +Because $s - N + 1 \ge t + 1$, +the same holds with $t + 1$ in place of $t$. + +Write $R$ for the product of these factors: + +$$R = \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}).$$ + +The product is empty, and $R$ is $1$, +when $n = t + N - 1$. +No factor involves $x_t$, +because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. +Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, +and then the cut above to each factor, +gives + +$$ +\begin{aligned} +p(x_t, \ldots, x_n) +&= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\ +&= p(x_t, \ldots, x_{t+N-1}) \, R. +\end{aligned} +$$ + +The same two steps, starting from $x_{t+1}$, give + +$$ +\begin{aligned} +p(x_{t+1}, \ldots, x_n) +&= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\ +&= p(x_{t+1}, \ldots, x_{t+N-1}) \, R. +\end{aligned} +$$ + +The left side of the second display is positive by assumption, +so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$. +By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob), + +$$ +\begin{aligned} +p(x_t \mid x_{t+1}, \ldots, x_n) +&= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\ +&= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\ +&= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\ +&= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}). +\end{aligned} +$$ + +The second line substitutes the two displays above, +the third cancels $R$, +and the fourth is the definition of conditional probability again. + +::: diff --git a/entropy.qmd b/entropy.qmd index 14946bdf..045877f8 100644 --- a/entropy.qmd +++ b/entropy.qmd @@ -418,6 +418,22 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*. {{< include ids/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd >}} +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-reversed-n-gram-model.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-reversed-n-gram-model.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd >}} + ## References {.unnumbered} ::: {#refs} From 1a5b7780a1e0fdd68e386039d77c6edcc1cb5a4b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 09:12:35 +0000 Subject: [PATCH 2/3] Give the 8.11 interpretation its own exercise; rename its pmfs The review on the backward PR noted that the last exercise asked two things (lab rule 7) and that the closing paragraph reused p, which the solution had defined as shorthand for a joint probability. The interpretation is now exr-reversed-text-information, and its pmfs are f (true) and g (the guesser's). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- ...xr-backward-n-gram-conditional-entropy.qmd | 3 --- _subfiles/_exr-reversed-text-information.qmd | 9 ++++++++ ...ol-backward-n-gram-conditional-entropy.qmd | 20 ---------------- _subfiles/_sol-reversed-text-information.qmd | 23 +++++++++++++++++++ entropy.qmd | 8 +++++++ 5 files changed, 40 insertions(+), 23 deletions(-) create mode 100644 _subfiles/_exr-reversed-text-information.qmd create mode 100644 _subfiles/_sol-reversed-text-information.qmd diff --git a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd index 597e1bb5..38b2e6bb 100644 --- a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd +++ b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd @@ -12,7 +12,4 @@ $$ = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. $$ -Together with @exr-reversed-text-total-entropy and @exr-forward-n-gram-conditional-entropy, -what does this say about the two guessing games in @rem-reversed-text-guessing? - ::: diff --git a/_subfiles/_exr-reversed-text-information.qmd b/_subfiles/_exr-reversed-text-information.qmd new file mode 100644 index 00000000..32901f29 --- /dev/null +++ b/_subfiles/_exr-reversed-text-information.qmd @@ -0,0 +1,9 @@ +::: {#exr-reversed-text-information} +#### Which guessing game is harder? + +Use @exr-reversed-text-total-entropy, @exr-forward-n-gram-conditional-entropy and @exr-backward-n-gram-conditional-entropy +to answer the question in @rem-reversed-text-guessing: +does a reversed text carry more information per character than the text read forward? +If not, why do people find the backward game harder? + +::: diff --git a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd index 0a47d0fc..bc6101df 100644 --- a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd +++ b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd @@ -41,24 +41,4 @@ $$ = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. $$ -This exercise and @exr-forward-n-gram-conditional-entropy give each character away from the ends of the text -the same [conditional entropy](#def-conditional-entropy) forward and backward, -and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward. -So the reversed text carries no more information per character than the text read forward. -The two games tie only on average: -in English, "q" is almost always followed by "u", -but "u" is rarely preceded by "q", -so a single guess can be easy one way and hard the other. - -People find the backward game harder -because their guesses are worse, -not because the text holds more information. -A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ -in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p$ -spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, -which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, -by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). -Years of reading forward make $q$ close to $p$ for forward guessing, -and no such practice makes it close backward. - ::: diff --git a/_subfiles/_sol-reversed-text-information.qmd b/_subfiles/_sol-reversed-text-information.qmd new file mode 100644 index 00000000..d3fe5794 --- /dev/null +++ b/_subfiles/_sol-reversed-text-information.qmd @@ -0,0 +1,23 @@ +::: {#sol-reversed-text-information} + +@exr-forward-n-gram-conditional-entropy and @exr-backward-n-gram-conditional-entropy give each character away from the ends of the text +the same [conditional entropy](#def-conditional-entropy) forward and backward, +and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward. +So the reversed text carries no more information per character than the text read forward. +The two games tie only on average: +in English, "q" is almost always followed by "u", +but "u" is rarely preceded by "q", +so a single guess can be easy one way and hard the other. + +People find the backward game harder +because their guesses are worse, +not because the text holds more information. +A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $g$ +in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $f$ +spends $\crossentropyf{f}{g}$ [bits](#def-bit) per character on average, +which exceeds $\entropyf{f}$ by $\kldivf{f}{g}$, +by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). +Years of reading forward make $g$ close to $f$ for forward guessing, +and no such practice makes it close backward. + +::: diff --git a/entropy.qmd b/entropy.qmd index 045877f8..c1de42cc 100644 --- a/entropy.qmd +++ b/entropy.qmd @@ -434,6 +434,14 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*. {{< include ids/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd >}} +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-reversed-text-information.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-reversed-text-information.qmd >}} + ## References {.unnumbered} ::: {#refs} From 4874dd901f87abb95f143bf2dffb6e18f21a05e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 09:15:05 +0000 Subject: [PATCH 3/3] Phrase the reversed-text exercise as one neutral question The review noted that "If not, why..." leaked the answer to the first question and split the exercise into two asks. The exercise now asks for one comparison and the explanation it supports. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- _subfiles/_exr-reversed-text-information.qmd | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/_subfiles/_exr-reversed-text-information.qmd b/_subfiles/_exr-reversed-text-information.qmd index 32901f29..2d957bee 100644 --- a/_subfiles/_exr-reversed-text-information.qmd +++ b/_subfiles/_exr-reversed-text-information.qmd @@ -2,8 +2,7 @@ #### Which guessing game is harder? Use @exr-reversed-text-total-entropy, @exr-forward-n-gram-conditional-entropy and @exr-backward-n-gram-conditional-entropy -to answer the question in @rem-reversed-text-guessing: -does a reversed text carry more information per character than the text read forward? -If not, why do people find the backward game harder? +to compare the information per character in a reversed text and in the text read forward, +and use that comparison to explain the difference in difficulty described in @rem-reversed-text-guessing. :::