Skip to content

fix(docs): pin the fumadocs-core peek omission that encodes markdown as entities - #281

Merged
os-bill merged 1 commit into
mainfrom
claude/issue-197-fumadocs-peek-patch
Sep 8, 2026
Merged

fix(docs): pin the fumadocs-core peek omission that encodes markdown as entities#281
os-bill merged 1 commit into
mainfrom
claude/issue-197-fumadocs-peek-patch

Conversation

@os-bill

@os-bill os-bill commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Fixes #197

Route A, as decided by the maintainer: pin the upstream one-liner with
pnpm.patchedDependencies and fix the producer, so /llms-full.txt and all 79
per-page /llms.mdx bodies are repaired together.

NOTATION. AMP stands for one literal ampersand character, per the card's
convention: this tracker's body sanitizer decodes HTML numeric character
references, including inside code fences. AMP#x2A; means the six characters
that would otherwise be typed as ampersand, hash, x, 2, A, semicolon.

What is patched

fumadocs-core, dist/mdx-plugins/stringifier.js. defaultStringifier wraps
every mdast-util-to-markdown serializer handler through modHandler and does
not carry the handler's peek onto the wrapper. containerPhrasing uses
Handle.peek to read the next sibling's first character cheaply and without
side effects; with it missing the lookahead invokes that sibling's full handler,
which leaves state.attentionEncodeSurroundingInfo set from node N+1 for the
loop to apply to node N. The patch copies .peek onto the wrapper.

The cause, the isolated A/B and the first end-to-end proof are on #197 and are
not repeated here. What this PR adds is the durable form of the same change and
the evidence that it survives a real pnpm install.

Measurements — re-taken in this tree, at base 50aacb9

Same harness before and after, reading the built bodies off
apps/docs/.next/server/app. Union re-run on the final tree at 5ce94d3.

before after
numeric character references, /llms-full.txt 67 0
numeric character references, 79 /llms.mdx bodies 67 0
malformed markdown link targets, of 661 4 0
absolute site targets in /llms-full.txt 560 562
line count of /llms-full.txt 12923 12923

Reference kinds before, in both outputs: AMP#x2A; x58, AMP#x60; x5,
AMP#x29; x4.

Three before-figures differ from the ones the decision comment quotes, and
the newer ones are the real ones.
Link targets are 661, not 650, and
absolute site targets 560, not 549 — eleven on-site links have landed in
content/docs/ since the investigation read 1ebd7b7. The malformed count (4)
and the reference counts (67, 58/5/4) are unchanged. The absolute-target figure
still moves by exactly +2: the two [AI Builder] links whose closing
parenthesis the bug had turned into AMP#x29;, which SITE_LINK in
llms-full.txt/route.ts could therefore never match.

Membership and order untouched. The 93 top-level # page headings are
byte-identical in sequence, all 799 heading lines are byte-identical in
sequence, and the line count is identical. Decoding the before body and diffing
against the after body leaves 21 differing lines: 19 are GFM table column
padding (a cell that loses 5 characters narrows its column, so every row in that
table is repadded) and 2 are the AI Builder links acquiring their absolute URL.
Nothing else differs.

The patch applies on a clean install

node_modules wiped in all three workspace projects, then
pnpm install --frozen-lockfile — which is what CI runs — exit 0. pnpm 10.28.2
prints no "patch applied" line on success, so the evidence is what the installer
itself wrote, read back off disk afterwards:

  • the virtual store directory it resolved to is
    node_modules/.pnpm/fumadocs-core@16.8.12_patch_hash=042796c0fe0ae66ccc2f1eadd4df4b8399a43c39c49facd3a0d938_.../node_modules/fumadocs-core
    — the patch_hash= segment is the installer's, not mine;
  • grep on that file, resolved through apps/docs' own module resolution,
    finds if (handler.peek) wrapped.peek = handler.peek; at line 81, and finds
    the pristine anchor line 0 times.

The patch step can fail — both ways

A patch step that cannot fail is indistinguishable from one that applied, so
both failure modes were demonstrated:

  1. Perturbed anchor, install refuses. One character changed in the patch's
    removed-context line (node.ctx to node.CTX, confirmed on disk by
    hash-object before and after), node_modules wiped, pnpm install exits
    1 with ERR_PNPM_PATCH_FAILED Could not apply patch ....
  2. Pin removed, the defect returns. pnpm.patchedDependencies deleted from
    package.json, node_modules wiped, install exit 0 — and the installed file
    carries the pristine anchor and no peek line. A full rebuild then puts the
    references back at exactly the baseline figures: 67 references, 4 of 661
    malformed, 695176 bytes, matching the before build byte count.

Both legs were restored from HEAD under an EXIT INT TERM trap and the
restore verified by git diff HEAD being empty, not by an exit code.

Where the manifest key went

pnpm 10.28.2 put patchedDependencies in the root package.json, under the
pnpm block that already carries onlyBuiltDependencies — not in
pnpm-workspace.yaml. That placement is pnpm patch-commit's own, left where
the installed tool wrote it and verified to work through a wiped
--frozen-lockfile install.

Worker size gate

Three wrangler deploy --dry-run readings in this container, all far under the
61440 KiB budget (headroom ~2895 KiB):

tree Total Upload:
pin removed (negative control) 58549.15 KiB
this branch, pnpm run build then package 58544.59 KiB
this branch, pnpm turbo run build then package 58544.37 KiB

The two readings of this same branch differ by 0.22 KiB, so a sub-KiB gap
between readings is not by itself evidence of a change in the bundle. No delta
is claimed here in either direction; the gate is green.

Gates run before pushing, on 5ce94d3

All exit 0, verdicts read from each command's own output:

  • pnpm install --frozen-lockfile on a wiped node_modules
  • node .github/scripts/check-node-floor.mjs --self-test and the real scan
  • node scripts/pm/check-half-states.mjs --self-test (1551 cases)
  • node apps/docs/scripts/gen-zh-hant.mjs --check (73 files byte-identical)
  • pnpm turbo run type-check --continue
  • pnpm turbo run build
  • node .github/scripts/check-locale-surface.mjs — 409 sitemap URLs, 0
    unexpected / 0 missing, and both llms bodies still carry all 60 en-only
    titles and none from other locales
  • pnpm turbo run test --force
  • opennextjs-cloudflare build --skipNextBuild plus the size weigh-in above
  • the pre-merge render check: smoke-docs.mjs against a local
    opennextjs-cloudflare preview, 4 pages rendered, its own negative control
    demonstrated red

Upstream, and the next bump

The defect is present in 16.8.12 (installed), 16.15.1, 16.15.2 and 16.15.8
(latest on 2026-09-08), each checked by unpacking the published tarball; the
anchor line is byte-identical in all four. Upgrading is not a fix. This session
cannot open an issue on fuma-nama/fumadocs, so a ready-to-post upstream report
is filed as a comment on #197 for the maintainer to route.

#239 bumps this package to 16.15.2. The pin is keyed to 16.8.12, so on any
other version the patch is simply not applied and the defect returns silently —
which is why the counts above should be re-measured on that PR, and the patch
regenerated against the new version in the same PR. 16.15.x also reformats an
if two hunks above this one in the same file. The patch file's header carries
all of this, plus the condition under which the file can be deleted.

Not touched

apps/docs/lib/source.ts, apps/docs/app/llms-full.txt/route.ts,
content/docs/, and .github/workflows/ci.yml (in flight on #277). No consumer
edit was needed, which is the result Route A was chosen for.

🤖 Generated with Claude Code

Generated by Claude Code


Generated by Claude Code

…as entities

`defaultStringifier` wraps every `mdast-util-to-markdown` serializer handler
through `modHandler` and does not carry the handler's `peek` onto the wrapper.
`containerPhrasing` uses `Handle.peek` to read the next sibling's first
character without side effects; with it missing, the lookahead invokes that
sibling's full handler and leaves `state.attentionEncodeSurroundingInfo` set
from node N+1, which the loop then applies to node N.

The effect reaches the served text: `/llms-full.txt` and all 79 per-page
`/llms.mdx` bodies carried 67 HTML numeric character references each, in files
that are not HTML, and 4 of 661 markdown link targets were delivered malformed
because the reference landed on the target's closing parenthesis. Two of those
were also invisible to the absolute-URL rewrite in `llms-full.txt/route.ts`,
whose `SITE_LINK` needs a literal closing parenthesis.

Fixed in the producer rather than by decoding on the way out: a consumer-side
decode would need a code-span guard, and the defect destroys inline-code closing
delimiters, so that guard would key on structure the bug has already broken.

The defect is present in 16.8.12, 16.15.1, 16.15.2 and 16.15.8, so upgrading is
not a fix. The patch file's header carries the upstream state it tracks and the
condition under which it can be deleted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S
@os-bill
os-bill marked this pull request as ready for review September 8, 2026 21:28
@os-bill
os-bill merged commit 6505ac1 into main Sep 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] llms-full.txt emits HTML numeric entities into a plain-text file, and two of them break the markdown link they sit in

2 participants