From 4c5c4a91bc69148c9fa63316f2dc5557cc1ff297 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 08:47:19 +0000 Subject: [PATCH 1/4] Add MacKay exercise 8.11: entropy of a text read backward Ezra asked for the worked solution to ITILA exercise 8.11 to live in ids. Under a stationary N-gram model, guessing a text backward costs the same average information per character as guessing it forward; the solution proves this step by step from the chain rule and explains why people still find the backward game harder. Stacked on the PR that defines stationary sequences and N-gram models. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- _subfiles/_exr-reversed-text-entropy.qmd | 50 ++++++ _subfiles/_sol-reversed-text-entropy.qmd | 217 +++++++++++++++++++++++ entropy.qmd | 8 + 3 files changed, 275 insertions(+) create mode 100644 _subfiles/_exr-reversed-text-entropy.qmd create mode 100644 _subfiles/_sol-reversed-text-entropy.qmd diff --git a/_subfiles/_exr-reversed-text-entropy.qmd b/_subfiles/_exr-reversed-text-entropy.qmd new file mode 100644 index 00000000..96d26f0c --- /dev/null +++ b/_subfiles/_exr-reversed-text-entropy.qmd @@ -0,0 +1,50 @@ +::: {#exr-reversed-text-entropy} +#### Predicting a text backward + +A reader can guess a text one character at a time from its start, +guessing each character from the characters already read. +The reader can also guess it from its end, +guessing each character from the characters that follow it. +Most people find the backward game harder, +so does a reversed text carry more information per character than the text read forward +[@mackay2003information, exercise 8.11]? + +Model the text as a [stationary sequence](#def-stationary-sequence) $X_1, X_2, \ldots$ of characters +that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, +and let $n \ge N$. + +1. Use the [chain rule for entropy](#thm-entropy-chain-rule-n), with the variables listed in each of two orders, to show that + + $$ + \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}} + = \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}, + $$ + + where the term for $t = 1$ on the left and the term for $t = n$ on the right are unconditional [entropies](https://morrison-lab.github.io/pds/expectation.html#def-entropy). + +2. For $N < t \le n$, show that the forward [conditional entropy](#def-conditional-entropy) of $X_t$ is + + $$ + \entropyf{X_t \mid X_1, \ldots, X_{t-1}} + = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. + $$ + +3. For $1 \le t \le n - N + 1$, show that given all the later characters, + $X_t$ depends only on the $N - 1$ characters just after it: + for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$, + + $$ + \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n) + = \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}). + $$ + +4. For $1 \le t \le n - N + 1$, show that the backward conditional entropy of $X_t$ is + + $$ + \entropyf{X_t \mid X_{t+1}, \ldots, X_n} + = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. + $$ + + What do parts 1, 2 and 4 say about the two guessing games? + +::: diff --git a/_subfiles/_sol-reversed-text-entropy.qmd b/_subfiles/_sol-reversed-text-entropy.qmd new file mode 100644 index 00000000..9cb92ced --- /dev/null +++ b/_subfiles/_sol-reversed-text-entropy.qmd @@ -0,0 +1,217 @@ +::: {#sol-reversed-text-entropy} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, +and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. +Each value's subscript says which variable takes it. + +1. Listing the same variables in another order does not change their [joint entropy](#def-joint-entropy), + because the reordered vector takes each reordered value with the same [probability](https://morrison-lab.github.io/pds/probability-basics.html#def-probability), + as in the proof of @cor-entropy-chain-rule-reversed. + For the same reason, + reordering the variables in a condition does not change a [conditional entropy](#def-conditional-entropy). + The [chain rule](#thm-entropy-chain-rule-n), with the variables in their usual order, gives + + $$\entropyf{X_1, \ldots, X_n} = \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}}.$$ + + With the variables listed as $X_n, X_{n-1}, \ldots, X_1$, + the $i$th variable is $X_{n+1-i}$, + and the variables before it are $X_n, \ldots, X_{n+2-i}$. + So + + $$ + \begin{aligned} + \entropyf{X_1, \ldots, X_n} + &= \entropyf{X_n, \ldots, X_1} \\ + &= \sum_{i=1}^{n} \entropyf{X_{n+1-i} \mid X_n, \ldots, X_{n+2-i}} \\ + &= \sum_{t=1}^{n} \entropyf{X_t \mid X_n, \ldots, X_{t+1}} \\ + &= \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}. + \end{aligned} + $$ + + The first line reorders the joint entropy, + the second is the chain rule, + the third substitutes $t = n + 1 - i$, + and the fourth reorders each condition. + Both sums equal $\entropyf{X_1, \ldots, X_n}$, + so they equal each other. + +2. Fix $t$ with $N < t \le n$. + For values $x_1, \ldots, x_{t-1}$ with $p(x_1, \ldots, x_{t-1}) > 0$, + @def-entropy-given-value makes $\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}}$ + a [function](https://morrison-lab.github.io/mcs/functions.html#def-function) of the [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p(\cdot \mid x_1, \ldots, x_{t-1})$ alone. + The [$N$-gram model](#def-n-gram-model) makes that conditional probability mass function equal to $p(\cdot \mid x_{t-N+1}, \ldots, x_{t-1})$, + so + + $$ + \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} + = \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}}. + $$ + + Then + + $$ + \begin{aligned} + \entropyf{X_t \mid X_1, \ldots, X_{t-1}} + &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} \\ + &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ + &= \sum_{x_{t-N+1}, \ldots, x_{t-1}} p(x_{t-N+1}, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ + &= \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. + \end{aligned} + $$ + + The first line is @thm-conditional-entropy-average, + with $(X_1, \ldots, X_{t-1})$ as the conditioning variable. + The second line uses the display above. + The third line sums over $x_1, \ldots, x_{t-N}$ first: + the second factor of each term does not depend on them, + and summing $p(x_1, \ldots, x_{t-1})$ over them gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t-N+1}, \ldots, x_{t-1})$. + The fourth line is @thm-conditional-entropy-average again. + + The two-variable [chain rule](#thm-entropy-chain-rule), + with $(X_{t-N+1}, \ldots, X_{t-1})$ as its first variable and $X_t$ as its second, + gives + + $$ + \entropyf{X_{t-N+1}, \ldots, X_t} + = \entropyf{X_{t-N+1}, \ldots, X_{t-1}} + \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. + $$ + + Subtracting $\entropyf{X_{t-N+1}, \ldots, X_{t-1}}$ from both sides gives + + $$ + \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}} + = \entropyf{X_{t-N+1}, \ldots, X_t} - \entropyf{X_{t-N+1}, \ldots, X_{t-1}}. + $$ + + An [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) depends only on the [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf). + The [stationary sequence](#def-stationary-sequence) with shift $s = t - N$ gives + $(X_{t-N+1}, \ldots, X_t)$ the same joint probability mass function as $(X_1, \ldots, X_N)$, + and $(X_{t-N+1}, \ldots, X_{t-1})$ the same one as $(X_1, \ldots, X_{N-1})$. + So + + $$ + \entropyf{X_t \mid X_1, \ldots, X_{t-1}} + = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. + $$ + +3. Fix $t$ with $1 \le t \le n - N + 1$. + First, for each $s$ with $t + N \le s \le n$, + the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$: + for values with $p(x_t, \ldots, x_{s-1}) > 0$, + + $$ + \begin{aligned} + p(x_s \mid x_t, \ldots, x_{s-1}) + &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\ + &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\ + &= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\ + &= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}). + \end{aligned} + $$ + + The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. + The second line is the [$N$-gram model](#def-n-gram-model), which applies because $s > N$. + The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. + The fourth line uses that a conditional probability mass function sums to $1$. + Because $s - N + 1 \ge t + 1$, + the same holds with $t + 1$ in place of $t$. + + Write $R$ for the product of these factors: + + $$R \eqdef \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}),$$ + + which is $1$ when $n = t + N - 1$. + No factor involves $x_t$, because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. + Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, + and then the cut above to each factor, + gives + + $$ + \begin{aligned} + p(x_t, \ldots, x_n) + &= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\ + &= p(x_t, \ldots, x_{t+N-1}) \, R. + \end{aligned} + $$ + + The same two steps, starting from $x_{t+1}$, give + + $$ + \begin{aligned} + p(x_{t+1}, \ldots, x_n) + &= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\ + &= p(x_{t+1}, \ldots, x_{t+N-1}) \, R. + \end{aligned} + $$ + + The left side of the second display is positive by assumption, + so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$. + By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob), + + $$ + \begin{aligned} + p(x_t \mid x_{t+1}, \ldots, x_n) + &= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\ + &= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\ + &= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\ + &= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}). + \end{aligned} + $$ + + The second line substitutes the two displays above, + the third cancels $R$, + and the fourth is the definition of conditional probability again. + +4. Fix $t$ with $1 \le t \le n - N + 1$. + The reasoning of part 2 carries over, + with the later characters in place of the earlier ones + and part 3 in place of the [$N$-gram model](#def-n-gram-model): + + $$ + \begin{aligned} + \entropyf{X_t \mid X_{t+1}, \ldots, X_n} + &= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\ + &= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ + &= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ + &= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}. + \end{aligned} + $$ + + The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable, + then gives + + $$ + \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}} + = \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}. + $$ + + The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives + $(X_t, \ldots, X_{t+N-1})$ the same joint probability mass function as $(X_1, \ldots, X_N)$, + and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$. + So + + $$ + \entropyf{X_t \mid X_{t+1}, \ldots, X_n} + = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. + $$ + + Parts 2 and 4 give each character away from the ends of the text + the same conditional entropy forward and backward, + and part 1 gives the whole text the same total forward and backward. + So the reversed text carries no more information per character than the text read forward. + The two games tie only on average: + in English, "q" is almost always followed by "u", + but "u" is rarely preceded by "q", + so a single guess can be easy one way and hard the other. + + People find the backward game harder because their guesses are worse, not because the text holds more information. + A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ + in place of the true conditional probability mass function $p$ + spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, + which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, + by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). + Years of reading forward make $q$ close to $p$ for forward guessing, + and no such practice makes it close backward. + +::: diff --git a/entropy.qmd b/entropy.qmd index a5ff5fac..4b0ba8e3 100644 --- a/entropy.qmd +++ b/entropy.qmd @@ -398,6 +398,14 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*. {{< include ids/_subfiles/_exm-not-n-gram-model.qmd >}} +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-reversed-text-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-reversed-text-entropy.qmd >}} + ## References {.unnumbered} ::: {#refs} From 43d720146b692cba518fa61cae7a47462ab3ce56 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 08:51:32 +0000 Subject: [PATCH 2/4] Break multi-clause lines in the 8.11 solution Same clause-level line-break pass the review asked for on the base PR. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- _subfiles/_sol-reversed-text-entropy.qmd | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/_subfiles/_sol-reversed-text-entropy.qmd b/_subfiles/_sol-reversed-text-entropy.qmd index 9cb92ced..94ccb9d0 100644 --- a/_subfiles/_sol-reversed-text-entropy.qmd +++ b/_subfiles/_sol-reversed-text-entropy.qmd @@ -111,7 +111,8 @@ Each value's subscript says which variable takes it. $$ The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. - The second line is the [$N$-gram model](#def-n-gram-model), which applies because $s > N$. + The second line is the [$N$-gram model](#def-n-gram-model), + which applies because $s > N$. The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. The fourth line uses that a conditional probability mass function sums to $1$. Because $s - N + 1 \ge t + 1$, @@ -122,7 +123,8 @@ Each value's subscript says which variable takes it. $$R \eqdef \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}),$$ which is $1$ when $n = t + N - 1$. - No factor involves $x_t$, because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. + No factor involves $x_t$, + because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, and then the cut above to each factor, gives @@ -205,7 +207,8 @@ Each value's subscript says which variable takes it. but "u" is rarely preceded by "q", so a single guess can be easy one way and hard the other. - People find the backward game harder because their guesses are worse, not because the text holds more information. + People find the backward game harder because their guesses are worse, + not because the text holds more information. A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ in place of the true conditional probability mass function $p$ spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, From 65365f94eac5c14da181697fd7a47a00a6d06cac Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 09:07:54 +0000 Subject: [PATCH 3/4] Split exercise 8.11 into four exercises with a motivating remark The Claude review asked for one exercise per part (lab rule 7). The guessing-game motivation and the MacKay citation move into a remark that the four exercises follow. Also breaks three multi-clause lines the review flagged and drops \eqdef for the solution's local abbreviation R. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- ...xr-backward-n-gram-conditional-entropy.qmd | 18 ++ ...exr-forward-n-gram-conditional-entropy.qmd | 14 ++ _subfiles/_exr-reversed-n-gram-model.qmd | 16 ++ _subfiles/_exr-reversed-text-entropy.qmd | 50 ---- .../_exr-reversed-text-total-entropy.qmd | 17 ++ _subfiles/_rem-reversed-text-guessing.qmd | 13 ++ ...ol-backward-n-gram-conditional-entropy.qmd | 64 +++++ ...sol-forward-n-gram-conditional-entropy.qmd | 66 ++++++ _subfiles/_sol-reversed-n-gram-model.qmd | 79 +++++++ _subfiles/_sol-reversed-text-entropy.qmd | 220 ------------------ .../_sol-reversed-text-total-entropy.qmd | 34 +++ entropy.qmd | 32 ++- 12 files changed, 351 insertions(+), 272 deletions(-) create mode 100644 _subfiles/_exr-backward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_exr-forward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_exr-reversed-n-gram-model.qmd delete mode 100644 _subfiles/_exr-reversed-text-entropy.qmd create mode 100644 _subfiles/_exr-reversed-text-total-entropy.qmd create mode 100644 _subfiles/_rem-reversed-text-guessing.qmd create mode 100644 _subfiles/_sol-backward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_sol-forward-n-gram-conditional-entropy.qmd create mode 100644 _subfiles/_sol-reversed-n-gram-model.qmd delete mode 100644 _subfiles/_sol-reversed-text-entropy.qmd create mode 100644 _subfiles/_sol-reversed-text-total-entropy.qmd diff --git a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..597e1bb5 --- /dev/null +++ b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd @@ -0,0 +1,18 @@ +::: {#exr-backward-n-gram-conditional-entropy} +#### Conditional entropy of an $N$-gram model read backward + +Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence) +that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, +let $n \ge N$, +and let $1 \le t \le n - N + 1$. +Use @exr-reversed-n-gram-model to show that the backward [conditional entropy](#def-conditional-entropy) of $X_t$ is + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +Together with @exr-reversed-text-total-entropy and @exr-forward-n-gram-conditional-entropy, +what does this say about the two guessing games in @rem-reversed-text-guessing? + +::: diff --git a/_subfiles/_exr-forward-n-gram-conditional-entropy.qmd b/_subfiles/_exr-forward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..87a47ba5 --- /dev/null +++ b/_subfiles/_exr-forward-n-gram-conditional-entropy.qmd @@ -0,0 +1,14 @@ +::: {#exr-forward-n-gram-conditional-entropy} +#### Conditional entropy of an $N$-gram model read forward + +Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence) +that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$. +For every $t > N$, +show that the forward [conditional entropy](#def-conditional-entropy) of $X_t$ is + +$$ +\entropyf{X_t \mid X_1, \ldots, X_{t-1}} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +::: diff --git a/_subfiles/_exr-reversed-n-gram-model.qmd b/_subfiles/_exr-reversed-n-gram-model.qmd new file mode 100644 index 00000000..adbf73ec --- /dev/null +++ b/_subfiles/_exr-reversed-n-gram-model.qmd @@ -0,0 +1,16 @@ +::: {#exr-reversed-n-gram-model} +#### An $N$-gram model read backward + +Let $X_1, X_2, \ldots$ be an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, +let $n \ge N$, +and let $1 \le t \le n - N + 1$. +Show that given all the later variables, +$X_t$ depends only on the $N - 1$ variables just after it: +for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$, + +$$ +\P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n) += \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}). +$$ + +::: diff --git a/_subfiles/_exr-reversed-text-entropy.qmd b/_subfiles/_exr-reversed-text-entropy.qmd deleted file mode 100644 index 96d26f0c..00000000 --- a/_subfiles/_exr-reversed-text-entropy.qmd +++ /dev/null @@ -1,50 +0,0 @@ -::: {#exr-reversed-text-entropy} -#### Predicting a text backward - -A reader can guess a text one character at a time from its start, -guessing each character from the characters already read. -The reader can also guess it from its end, -guessing each character from the characters that follow it. -Most people find the backward game harder, -so does a reversed text carry more information per character than the text read forward -[@mackay2003information, exercise 8.11]? - -Model the text as a [stationary sequence](#def-stationary-sequence) $X_1, X_2, \ldots$ of characters -that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, -and let $n \ge N$. - -1. Use the [chain rule for entropy](#thm-entropy-chain-rule-n), with the variables listed in each of two orders, to show that - - $$ - \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}} - = \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}, - $$ - - where the term for $t = 1$ on the left and the term for $t = n$ on the right are unconditional [entropies](https://morrison-lab.github.io/pds/expectation.html#def-entropy). - -2. For $N < t \le n$, show that the forward [conditional entropy](#def-conditional-entropy) of $X_t$ is - - $$ - \entropyf{X_t \mid X_1, \ldots, X_{t-1}} - = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. - $$ - -3. For $1 \le t \le n - N + 1$, show that given all the later characters, - $X_t$ depends only on the $N - 1$ characters just after it: - for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$, - - $$ - \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n) - = \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}). - $$ - -4. For $1 \le t \le n - N + 1$, show that the backward conditional entropy of $X_t$ is - - $$ - \entropyf{X_t \mid X_{t+1}, \ldots, X_n} - = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. - $$ - - What do parts 1, 2 and 4 say about the two guessing games? - -::: diff --git a/_subfiles/_exr-reversed-text-total-entropy.qmd b/_subfiles/_exr-reversed-text-total-entropy.qmd new file mode 100644 index 00000000..04bf1063 --- /dev/null +++ b/_subfiles/_exr-reversed-text-total-entropy.qmd @@ -0,0 +1,17 @@ +::: {#exr-reversed-text-total-entropy} +#### The whole text costs the same in both orders + +Let $X_1, \ldots, X_n$ be [discrete random variables](https://morrison-lab.github.io/pds/random-variables.html#def-discrete-rv). +Use the [chain rule for entropy](#thm-entropy-chain-rule-n), +with the variables listed in each of two orders, +to show that + +$$ +\sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}} += \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}, +$$ + +where the term for $t = 1$ on the left and the term for $t = n$ on the right +are unconditional [entropies](https://morrison-lab.github.io/pds/expectation.html#def-entropy). + +::: diff --git a/_subfiles/_rem-reversed-text-guessing.qmd b/_subfiles/_rem-reversed-text-guessing.qmd new file mode 100644 index 00000000..684a30b8 --- /dev/null +++ b/_subfiles/_rem-reversed-text-guessing.qmd @@ -0,0 +1,13 @@ +:::{#rem-reversed-text-guessing} +#### Guessing a text backward + +A reader can guess a text one character at a time from its start, +guessing each character from the characters already read. +The reader can also guess it from its end, +guessing each character from the characters that follow it. +Most people find the backward game harder. +That raises a question: +does a reversed text carry more information per character than the text read forward +[@mackay2003information, exercise 8.11]? + +::: diff --git a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..0a47d0fc --- /dev/null +++ b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd @@ -0,0 +1,64 @@ +::: {#sol-backward-n-gram-conditional-entropy} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$. +Each value's subscript says which variable takes it. + +The reasoning of @sol-forward-n-gram-conditional-entropy carries over, +with the later variables in place of the earlier ones +and @exr-reversed-n-gram-model in place of the [$N$-gram model](#def-n-gram-model): + +$$ +\begin{aligned} +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} +&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\ +&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ +&= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ +&= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}. +\end{aligned} +$$ + +The first and fourth lines are @thm-conditional-entropy-average. +The second line uses @exr-reversed-n-gram-model and @def-entropy-given-value. +The third line sums over $x_{t+N}, \ldots, x_n$ first, +which gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t+1}, \ldots, x_{t+N-1})$. + +The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable, +then gives + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}} += \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}. +$$ + +The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives +$(X_t, \ldots, X_{t+N-1})$ the same [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf) as $(X_1, \ldots, X_N)$, +and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$. +So + +$$ +\entropyf{X_t \mid X_{t+1}, \ldots, X_n} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +This exercise and @exr-forward-n-gram-conditional-entropy give each character away from the ends of the text +the same [conditional entropy](#def-conditional-entropy) forward and backward, +and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward. +So the reversed text carries no more information per character than the text read forward. +The two games tie only on average: +in English, "q" is almost always followed by "u", +but "u" is rarely preceded by "q", +so a single guess can be easy one way and hard the other. + +People find the backward game harder +because their guesses are worse, +not because the text holds more information. +A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ +in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p$ +spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, +which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, +by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). +Years of reading forward make $q$ close to $p$ for forward guessing, +and no such practice makes it close backward. + +::: diff --git a/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd b/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd new file mode 100644 index 00000000..bfaca4c9 --- /dev/null +++ b/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd @@ -0,0 +1,66 @@ +::: {#sol-forward-n-gram-conditional-entropy} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, +and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. +Each value's subscript says which variable takes it. + +For values $x_1, \ldots, x_{t-1}$ with $p(x_1, \ldots, x_{t-1}) > 0$, +@def-entropy-given-value makes $\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}}$ +a [function](https://morrison-lab.github.io/mcs/functions.html#def-function) of the [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p(\cdot \mid x_1, \ldots, x_{t-1})$ alone. +The [$N$-gram model](#def-n-gram-model) makes that conditional probability mass function equal to $p(\cdot \mid x_{t-N+1}, \ldots, x_{t-1})$, +so + +$$ +\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} += \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}}. +$$ + +Then + +$$ +\begin{aligned} +\entropyf{X_t \mid X_1, \ldots, X_{t-1}} +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} \\ +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ +&= \sum_{x_{t-N+1}, \ldots, x_{t-1}} p(x_{t-N+1}, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ +&= \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. +\end{aligned} +$$ + +The first line is @thm-conditional-entropy-average, +with $(X_1, \ldots, X_{t-1})$ as the conditioning variable. +The second line uses the display above. +The third line sums over $x_1, \ldots, x_{t-N}$ first: +the second factor of each term does not depend on them, +and summing $p(x_1, \ldots, x_{t-1})$ over them gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t-N+1}, \ldots, x_{t-1})$. +The fourth line is @thm-conditional-entropy-average again. + +The two-variable [chain rule](#thm-entropy-chain-rule), +with $(X_{t-N+1}, \ldots, X_{t-1})$ as its first variable and $X_t$ as its second, +gives + +$$ +\entropyf{X_{t-N+1}, \ldots, X_t} += \entropyf{X_{t-N+1}, \ldots, X_{t-1}} + \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. +$$ + +Subtracting $\entropyf{X_{t-N+1}, \ldots, X_{t-1}}$ from both sides gives + +$$ +\entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}} += \entropyf{X_{t-N+1}, \ldots, X_t} - \entropyf{X_{t-N+1}, \ldots, X_{t-1}}. +$$ + +An [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) depends only on the [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf). +The [stationary sequence](#def-stationary-sequence) with shift $s = t - N$ gives +$(X_{t-N+1}, \ldots, X_t)$ the same joint probability mass function as $(X_1, \ldots, X_N)$, +and $(X_{t-N+1}, \ldots, X_{t-1})$ the same one as $(X_1, \ldots, X_{N-1})$. +So + +$$ +\entropyf{X_t \mid X_1, \ldots, X_{t-1}} += \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. +$$ + +::: diff --git a/_subfiles/_sol-reversed-n-gram-model.qmd b/_subfiles/_sol-reversed-n-gram-model.qmd new file mode 100644 index 00000000..a7c9775a --- /dev/null +++ b/_subfiles/_sol-reversed-n-gram-model.qmd @@ -0,0 +1,79 @@ +::: {#sol-reversed-n-gram-model} + +To shorten the displays, +write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, +and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. +Each value's subscript says which variable takes it. + +First, for each $s$ with $t + N \le s \le n$, +the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$: +for values with $p(x_t, \ldots, x_{s-1}) > 0$, + +$$ +\begin{aligned} +p(x_s \mid x_t, \ldots, x_{s-1}) +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\ +&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\ +&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\ +&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}). +\end{aligned} +$$ + +The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. +The second line is the [$N$-gram model](#def-n-gram-model), +which applies +because $s > N$. +The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. +The fourth line uses that a [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) sums to $1$. +Because $s - N + 1 \ge t + 1$, +the same holds with $t + 1$ in place of $t$. + +Write $R$ for the product of these factors: + +$$R = \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}).$$ + +The product is empty, and $R$ is $1$, +when $n = t + N - 1$. +No factor involves $x_t$, +because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. +Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, +and then the cut above to each factor, +gives + +$$ +\begin{aligned} +p(x_t, \ldots, x_n) +&= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\ +&= p(x_t, \ldots, x_{t+N-1}) \, R. +\end{aligned} +$$ + +The same two steps, starting from $x_{t+1}$, give + +$$ +\begin{aligned} +p(x_{t+1}, \ldots, x_n) +&= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\ +&= p(x_{t+1}, \ldots, x_{t+N-1}) \, R. +\end{aligned} +$$ + +The left side of the second display is positive by assumption, +so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$. +By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob), + +$$ +\begin{aligned} +p(x_t \mid x_{t+1}, \ldots, x_n) +&= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\ +&= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\ +&= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\ +&= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}). +\end{aligned} +$$ + +The second line substitutes the two displays above, +the third cancels $R$, +and the fourth is the definition of conditional probability again. + +::: diff --git a/_subfiles/_sol-reversed-text-entropy.qmd b/_subfiles/_sol-reversed-text-entropy.qmd deleted file mode 100644 index 94ccb9d0..00000000 --- a/_subfiles/_sol-reversed-text-entropy.qmd +++ /dev/null @@ -1,220 +0,0 @@ -::: {#sol-reversed-text-entropy} - -To shorten the displays, -write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, -and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. -Each value's subscript says which variable takes it. - -1. Listing the same variables in another order does not change their [joint entropy](#def-joint-entropy), - because the reordered vector takes each reordered value with the same [probability](https://morrison-lab.github.io/pds/probability-basics.html#def-probability), - as in the proof of @cor-entropy-chain-rule-reversed. - For the same reason, - reordering the variables in a condition does not change a [conditional entropy](#def-conditional-entropy). - The [chain rule](#thm-entropy-chain-rule-n), with the variables in their usual order, gives - - $$\entropyf{X_1, \ldots, X_n} = \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}}.$$ - - With the variables listed as $X_n, X_{n-1}, \ldots, X_1$, - the $i$th variable is $X_{n+1-i}$, - and the variables before it are $X_n, \ldots, X_{n+2-i}$. - So - - $$ - \begin{aligned} - \entropyf{X_1, \ldots, X_n} - &= \entropyf{X_n, \ldots, X_1} \\ - &= \sum_{i=1}^{n} \entropyf{X_{n+1-i} \mid X_n, \ldots, X_{n+2-i}} \\ - &= \sum_{t=1}^{n} \entropyf{X_t \mid X_n, \ldots, X_{t+1}} \\ - &= \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}. - \end{aligned} - $$ - - The first line reorders the joint entropy, - the second is the chain rule, - the third substitutes $t = n + 1 - i$, - and the fourth reorders each condition. - Both sums equal $\entropyf{X_1, \ldots, X_n}$, - so they equal each other. - -2. Fix $t$ with $N < t \le n$. - For values $x_1, \ldots, x_{t-1}$ with $p(x_1, \ldots, x_{t-1}) > 0$, - @def-entropy-given-value makes $\entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}}$ - a [function](https://morrison-lab.github.io/mcs/functions.html#def-function) of the [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p(\cdot \mid x_1, \ldots, x_{t-1})$ alone. - The [$N$-gram model](#def-n-gram-model) makes that conditional probability mass function equal to $p(\cdot \mid x_{t-N+1}, \ldots, x_{t-1})$, - so - - $$ - \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} - = \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}}. - $$ - - Then - - $$ - \begin{aligned} - \entropyf{X_t \mid X_1, \ldots, X_{t-1}} - &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_1 = x_1, \ldots, X_{t-1} = x_{t-1}} \\ - &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ - &= \sum_{x_{t-N+1}, \ldots, x_{t-1}} p(x_{t-N+1}, \ldots, x_{t-1}) \, \entropyf{X_t \mid X_{t-N+1} = x_{t-N+1}, \ldots, X_{t-1} = x_{t-1}} \\ - &= \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. - \end{aligned} - $$ - - The first line is @thm-conditional-entropy-average, - with $(X_1, \ldots, X_{t-1})$ as the conditioning variable. - The second line uses the display above. - The third line sums over $x_1, \ldots, x_{t-N}$ first: - the second factor of each term does not depend on them, - and summing $p(x_1, \ldots, x_{t-1})$ over them gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t-N+1}, \ldots, x_{t-1})$. - The fourth line is @thm-conditional-entropy-average again. - - The two-variable [chain rule](#thm-entropy-chain-rule), - with $(X_{t-N+1}, \ldots, X_{t-1})$ as its first variable and $X_t$ as its second, - gives - - $$ - \entropyf{X_{t-N+1}, \ldots, X_t} - = \entropyf{X_{t-N+1}, \ldots, X_{t-1}} + \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}}. - $$ - - Subtracting $\entropyf{X_{t-N+1}, \ldots, X_{t-1}}$ from both sides gives - - $$ - \entropyf{X_t \mid X_{t-N+1}, \ldots, X_{t-1}} - = \entropyf{X_{t-N+1}, \ldots, X_t} - \entropyf{X_{t-N+1}, \ldots, X_{t-1}}. - $$ - - An [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) depends only on the [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf). - The [stationary sequence](#def-stationary-sequence) with shift $s = t - N$ gives - $(X_{t-N+1}, \ldots, X_t)$ the same joint probability mass function as $(X_1, \ldots, X_N)$, - and $(X_{t-N+1}, \ldots, X_{t-1})$ the same one as $(X_1, \ldots, X_{N-1})$. - So - - $$ - \entropyf{X_t \mid X_1, \ldots, X_{t-1}} - = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. - $$ - -3. Fix $t$ with $1 \le t \le n - N + 1$. - First, for each $s$ with $t + N \le s \le n$, - the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$: - for values with $p(x_t, \ldots, x_{s-1}) > 0$, - - $$ - \begin{aligned} - p(x_s \mid x_t, \ldots, x_{s-1}) - &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\ - &= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\ - &= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\ - &= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}). - \end{aligned} - $$ - - The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. - The second line is the [$N$-gram model](#def-n-gram-model), - which applies because $s > N$. - The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. - The fourth line uses that a conditional probability mass function sums to $1$. - Because $s - N + 1 \ge t + 1$, - the same holds with $t + 1$ in place of $t$. - - Write $R$ for the product of these factors: - - $$R \eqdef \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}),$$ - - which is $1$ when $n = t + N - 1$. - No factor involves $x_t$, - because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. - Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, - and then the cut above to each factor, - gives - - $$ - \begin{aligned} - p(x_t, \ldots, x_n) - &= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\ - &= p(x_t, \ldots, x_{t+N-1}) \, R. - \end{aligned} - $$ - - The same two steps, starting from $x_{t+1}$, give - - $$ - \begin{aligned} - p(x_{t+1}, \ldots, x_n) - &= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\ - &= p(x_{t+1}, \ldots, x_{t+N-1}) \, R. - \end{aligned} - $$ - - The left side of the second display is positive by assumption, - so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$. - By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob), - - $$ - \begin{aligned} - p(x_t \mid x_{t+1}, \ldots, x_n) - &= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\ - &= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\ - &= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\ - &= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}). - \end{aligned} - $$ - - The second line substitutes the two displays above, - the third cancels $R$, - and the fourth is the definition of conditional probability again. - -4. Fix $t$ with $1 \le t \le n - N + 1$. - The reasoning of part 2 carries over, - with the later characters in place of the earlier ones - and part 3 in place of the [$N$-gram model](#def-n-gram-model): - - $$ - \begin{aligned} - \entropyf{X_t \mid X_{t+1}, \ldots, X_n} - &= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\ - &= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ - &= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ - &= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}. - \end{aligned} - $$ - - The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable, - then gives - - $$ - \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}} - = \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}. - $$ - - The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives - $(X_t, \ldots, X_{t+N-1})$ the same joint probability mass function as $(X_1, \ldots, X_N)$, - and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$. - So - - $$ - \entropyf{X_t \mid X_{t+1}, \ldots, X_n} - = \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. - $$ - - Parts 2 and 4 give each character away from the ends of the text - the same conditional entropy forward and backward, - and part 1 gives the whole text the same total forward and backward. - So the reversed text carries no more information per character than the text read forward. - The two games tie only on average: - in English, "q" is almost always followed by "u", - but "u" is rarely preceded by "q", - so a single guess can be easy one way and hard the other. - - People find the backward game harder because their guesses are worse, - not because the text holds more information. - A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ - in place of the true conditional probability mass function $p$ - spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, - which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, - by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). - Years of reading forward make $q$ close to $p$ for forward guessing, - and no such practice makes it close backward. - -::: diff --git a/_subfiles/_sol-reversed-text-total-entropy.qmd b/_subfiles/_sol-reversed-text-total-entropy.qmd new file mode 100644 index 00000000..c2f33a0a --- /dev/null +++ b/_subfiles/_sol-reversed-text-total-entropy.qmd @@ -0,0 +1,34 @@ +::: {#sol-reversed-text-total-entropy} + +Listing the same variables in another order does not change their [joint entropy](#def-joint-entropy), +because the reordered vector takes each reordered value with the same [probability](https://morrison-lab.github.io/pds/probability-basics.html#def-probability), +as in the proof of @cor-entropy-chain-rule-reversed. +For the same reason, +reordering the variables in a condition does not change a [conditional entropy](#def-conditional-entropy). +The [chain rule](#thm-entropy-chain-rule-n), with the variables in their usual order, gives + +$$\entropyf{X_1, \ldots, X_n} = \sum_{t=1}^{n} \entropyf{X_t \mid X_1, \ldots, X_{t-1}}.$$ + +With the variables listed as $X_n, X_{n-1}, \ldots, X_1$, +the $i$th variable is $X_{n+1-i}$, +and the variables before it are $X_n, \ldots, X_{n+2-i}$. +So + +$$ +\begin{aligned} +\entropyf{X_1, \ldots, X_n} +&= \entropyf{X_n, \ldots, X_1} \\ +&= \sum_{i=1}^{n} \entropyf{X_{n+1-i} \mid X_n, \ldots, X_{n+2-i}} \\ +&= \sum_{t=1}^{n} \entropyf{X_t \mid X_n, \ldots, X_{t+1}} \\ +&= \sum_{t=1}^{n} \entropyf{X_t \mid X_{t+1}, \ldots, X_n}. +\end{aligned} +$$ + +The first line reorders the joint entropy, +the second is the chain rule, +the third substitutes $t = n + 1 - i$, +and the fourth reorders each condition. +Both sums equal $\entropyf{X_1, \ldots, X_n}$, +so they equal each other. + +::: diff --git a/entropy.qmd b/entropy.qmd index 4b0ba8e3..045877f8 100644 --- a/entropy.qmd +++ b/entropy.qmd @@ -400,11 +400,39 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*. {{< slidebreak >}} -{{< include ids/_subfiles/_exr-reversed-text-entropy.qmd >}} +{{< include ids/_subfiles/_rem-reversed-text-guessing.qmd >}} {{< slidebreak >}} -{{< include ids/_subfiles/_sol-reversed-text-entropy.qmd >}} +{{< include ids/_subfiles/_exr-reversed-text-total-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-reversed-text-total-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-forward-n-gram-conditional-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-reversed-n-gram-model.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-reversed-n-gram-model.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd >}} + +{{< slidebreak >}} + +{{< include ids/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd >}} ## References {.unnumbered} From ccca005e78d3b416aa8637925c4835259b235b5a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 09:08:07 +0000 Subject: [PATCH 4/4] Move the backward half of exercise 8.11 to its own stacked PR Splitting the exercise into four put this PR over the lab's 300-line cap. This PR keeps the remark and the first two exercises (both orders give the same total, and the forward conditional entropy). The backward exercises follow in a PR stacked on this one. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019xfc9pXuEzzVRdTKMWnCh8 --- ...xr-backward-n-gram-conditional-entropy.qmd | 18 ----- _subfiles/_exr-reversed-n-gram-model.qmd | 16 ---- ...ol-backward-n-gram-conditional-entropy.qmd | 64 --------------- _subfiles/_sol-reversed-n-gram-model.qmd | 79 ------------------- entropy.qmd | 16 ---- 5 files changed, 193 deletions(-) delete mode 100644 _subfiles/_exr-backward-n-gram-conditional-entropy.qmd delete mode 100644 _subfiles/_exr-reversed-n-gram-model.qmd delete mode 100644 _subfiles/_sol-backward-n-gram-conditional-entropy.qmd delete mode 100644 _subfiles/_sol-reversed-n-gram-model.qmd diff --git a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd b/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd deleted file mode 100644 index 597e1bb5..00000000 --- a/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd +++ /dev/null @@ -1,18 +0,0 @@ -::: {#exr-backward-n-gram-conditional-entropy} -#### Conditional entropy of an $N$-gram model read backward - -Let $X_1, X_2, \ldots$ be a [stationary sequence](#def-stationary-sequence) -that is also an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, -let $n \ge N$, -and let $1 \le t \le n - N + 1$. -Use @exr-reversed-n-gram-model to show that the backward [conditional entropy](#def-conditional-entropy) of $X_t$ is - -$$ -\entropyf{X_t \mid X_{t+1}, \ldots, X_n} -= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. -$$ - -Together with @exr-reversed-text-total-entropy and @exr-forward-n-gram-conditional-entropy, -what does this say about the two guessing games in @rem-reversed-text-guessing? - -::: diff --git a/_subfiles/_exr-reversed-n-gram-model.qmd b/_subfiles/_exr-reversed-n-gram-model.qmd deleted file mode 100644 index adbf73ec..00000000 --- a/_subfiles/_exr-reversed-n-gram-model.qmd +++ /dev/null @@ -1,16 +0,0 @@ -::: {#exr-reversed-n-gram-model} -#### An $N$-gram model read backward - -Let $X_1, X_2, \ldots$ be an [$N$-gram model](#def-n-gram-model) with $N \ge 2$, -let $n \ge N$, -and let $1 \le t \le n - N + 1$. -Show that given all the later variables, -$X_t$ depends only on the $N - 1$ variables just after it: -for every choice of values $x_t, \ldots, x_n$ with $\P(X_{t+1} = x_{t+1}, \ldots, X_n = x_n) > 0$, - -$$ -\P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n) -= \P(X_t = x_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}). -$$ - -::: diff --git a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd b/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd deleted file mode 100644 index 0a47d0fc..00000000 --- a/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd +++ /dev/null @@ -1,64 +0,0 @@ -::: {#sol-backward-n-gram-conditional-entropy} - -To shorten the displays, -write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$. -Each value's subscript says which variable takes it. - -The reasoning of @sol-forward-n-gram-conditional-entropy carries over, -with the later variables in place of the earlier ones -and @exr-reversed-n-gram-model in place of the [$N$-gram model](#def-n-gram-model): - -$$ -\begin{aligned} -\entropyf{X_t \mid X_{t+1}, \ldots, X_n} -&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_n = x_n} \\ -&= \sum_{x_{t+1}, \ldots, x_n} p(x_{t+1}, \ldots, x_n) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ -&= \sum_{x_{t+1}, \ldots, x_{t+N-1}} p(x_{t+1}, \ldots, x_{t+N-1}) \, \entropyf{X_t \mid X_{t+1} = x_{t+1}, \ldots, X_{t+N-1} = x_{t+N-1}} \\ -&= \entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}}. -\end{aligned} -$$ - -The first and fourth lines are @thm-conditional-entropy-average. -The second line uses @exr-reversed-n-gram-model and @def-entropy-given-value. -The third line sums over $x_{t+N}, \ldots, x_n$ first, -which gives the [marginal](https://morrison-lab.github.io/pds/random-variables.html#def-marginal) $p(x_{t+1}, \ldots, x_{t+N-1})$. - -The [chain rule](#thm-entropy-chain-rule), with $(X_{t+1}, \ldots, X_{t+N-1})$ as its first variable, -then gives - -$$ -\entropyf{X_t \mid X_{t+1}, \ldots, X_{t+N-1}} -= \entropyf{X_t, \ldots, X_{t+N-1}} - \entropyf{X_{t+1}, \ldots, X_{t+N-1}}. -$$ - -The [stationary sequence](#def-stationary-sequence) with shift $s = t - 1$ gives -$(X_t, \ldots, X_{t+N-1})$ the same [joint probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-joint-pmf) as $(X_1, \ldots, X_N)$, -and with shift $s = t$ it gives $(X_{t+1}, \ldots, X_{t+N-1})$ the same one as $(X_1, \ldots, X_{N-1})$. -So - -$$ -\entropyf{X_t \mid X_{t+1}, \ldots, X_n} -= \entropyf{X_1, \ldots, X_N} - \entropyf{X_1, \ldots, X_{N-1}}. -$$ - -This exercise and @exr-forward-n-gram-conditional-entropy give each character away from the ends of the text -the same [conditional entropy](#def-conditional-entropy) forward and backward, -and @exr-reversed-text-total-entropy gives the whole text the same sum of conditional entropies forward and backward. -So the reversed text carries no more information per character than the text read forward. -The two games tie only on average: -in English, "q" is almost always followed by "u", -but "u" is rarely preceded by "q", -so a single guess can be easy one way and hard the other. - -People find the backward game harder -because their guesses are worse, -not because the text holds more information. -A reader who guesses with a [probability mass function](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) $q$ -in place of the true [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) $p$ -spends $\crossentropyf{p}{q}$ [bits](#def-bit) per character on average, -which exceeds $\entropyf{p}$ by $\kldivf{p}{q}$, -by the [decomposition of cross-entropy](source-coding.qmd#thm-cross-entropy-decomposition). -Years of reading forward make $q$ close to $p$ for forward guessing, -and no such practice makes it close backward. - -::: diff --git a/_subfiles/_sol-reversed-n-gram-model.qmd b/_subfiles/_sol-reversed-n-gram-model.qmd deleted file mode 100644 index a7c9775a..00000000 --- a/_subfiles/_sol-reversed-n-gram-model.qmd +++ /dev/null @@ -1,79 +0,0 @@ -::: {#sol-reversed-n-gram-model} - -To shorten the displays, -write $p(x_a, \ldots, x_b)$ for $\P(X_a = x_a, \ldots, X_b = x_b)$, -and $p(x_a \mid x_c, \ldots, x_d)$ for $\P(X_a = x_a \mid X_c = x_c, \ldots, X_d = x_d)$. -Each value's subscript says which variable takes it. - -First, for each $s$ with $t + N \le s \le n$, -the condition on $X_t, \ldots, X_{s-1}$ can be cut to the $N - 1$ variables just before $X_s$: -for values with $p(x_t, \ldots, x_{s-1}) > 0$, - -$$ -\begin{aligned} -p(x_s \mid x_t, \ldots, x_{s-1}) -&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_1, \ldots, x_{s-1}) \\ -&= \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \, p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \\ -&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}) \sum_{x_1, \ldots, x_{t-1}} p(x_1, \ldots, x_{t-1} \mid x_t, \ldots, x_{s-1}) \\ -&= p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}). -\end{aligned} -$$ - -The first line is the [law of total probability](https://morrison-lab.github.io/pds/probability-basics.html#thm-total-prob-conditional), given $X_t, \ldots, X_{s-1}$. -The second line is the [$N$-gram model](#def-n-gram-model), -which applies -because $s > N$. -The third line takes the factor that does not depend on $x_1, \ldots, x_{t-1}$ out of the sum. -The fourth line uses that a [conditional probability mass function](https://morrison-lab.github.io/pds/expectation.html#def-cond-pmf) sums to $1$. -Because $s - N + 1 \ge t + 1$, -the same holds with $t + 1$ in place of $t$. - -Write $R$ for the product of these factors: - -$$R = \prod_{s=t+N}^{n} p(x_s \mid x_{s-N+1}, \ldots, x_{s-1}).$$ - -The product is empty, and $R$ is $1$, -when $n = t + N - 1$. -No factor involves $x_t$, -because each starts at $x_{s-N+1}$ with $s - N + 1 \ge t + 1$. -Applying the [multiplication rule](https://morrison-lab.github.io/pds/probability-basics.html#thm-law-conditional-prob) once for each $s$ from $t + N$ to $n$, -and then the cut above to each factor, -gives - -$$ -\begin{aligned} -p(x_t, \ldots, x_n) -&= p(x_t, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_t, \ldots, x_{s-1}) \\ -&= p(x_t, \ldots, x_{t+N-1}) \, R. -\end{aligned} -$$ - -The same two steps, starting from $x_{t+1}$, give - -$$ -\begin{aligned} -p(x_{t+1}, \ldots, x_n) -&= p(x_{t+1}, \ldots, x_{t+N-1}) \prod_{s=t+N}^{n} p(x_s \mid x_{t+1}, \ldots, x_{s-1}) \\ -&= p(x_{t+1}, \ldots, x_{t+N-1}) \, R. -\end{aligned} -$$ - -The left side of the second display is positive by assumption, -so $R > 0$ and $p(x_{t+1}, \ldots, x_{t+N-1}) > 0$. -By the definition of [conditional probability](https://morrison-lab.github.io/pds/probability-basics.html#def-conditional-prob), - -$$ -\begin{aligned} -p(x_t \mid x_{t+1}, \ldots, x_n) -&= \frac{p(x_t, \ldots, x_n)}{p(x_{t+1}, \ldots, x_n)} \\ -&= \frac{p(x_t, \ldots, x_{t+N-1}) \, R}{p(x_{t+1}, \ldots, x_{t+N-1}) \, R} \\ -&= \frac{p(x_t, \ldots, x_{t+N-1})}{p(x_{t+1}, \ldots, x_{t+N-1})} \\ -&= p(x_t \mid x_{t+1}, \ldots, x_{t+N-1}). -\end{aligned} -$$ - -The second line substitutes the two displays above, -the third cancels $R$, -and the fourth is the definition of conditional probability again. - -::: diff --git a/entropy.qmd b/entropy.qmd index 045877f8..14946bdf 100644 --- a/entropy.qmd +++ b/entropy.qmd @@ -418,22 +418,6 @@ in *[Probability for Data Science](https://morrison-lab.github.io/pds/)*. {{< include ids/_subfiles/_sol-forward-n-gram-conditional-entropy.qmd >}} -{{< slidebreak >}} - -{{< include ids/_subfiles/_exr-reversed-n-gram-model.qmd >}} - -{{< slidebreak >}} - -{{< include ids/_subfiles/_sol-reversed-n-gram-model.qmd >}} - -{{< slidebreak >}} - -{{< include ids/_subfiles/_exr-backward-n-gram-conditional-entropy.qmd >}} - -{{< slidebreak >}} - -{{< include ids/_subfiles/_sol-backward-n-gram-conditional-entropy.qmd >}} - ## References {.unnumbered} ::: {#refs}