Skip to content

fix(docs-audit): a route anchor must name the route the page names, and a row says whether it came from prose - #18515

Merged
os-try-charles merged 2 commits into
mainfrom
claude/issue-16696-docs-drift-anchor-precision
Sep 16, 2026
Merged

os-try-charles merged 2 commits into
mainfrom
claude/issue-16696-docs-drift-anchor-precision

Conversation

@os-try-charles

@os-try-charles os-try-charles commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #16696.

The docs-drift bot's contract is that a row is reportable: "Each row says which anchor put it there, so a wrong row is reportable rather than merely annoying." #16696 read all nine hand-written rows on PR #16694 by hand and reported three shapes. Triage (comment 5578416342) ruled a different disposition for each, and this PR keeps them apart.


Finding 1 — the diagnosis first: this is MATCH WIDENING, not mis-attribution

Triage's hard precondition: 「⛔ 修之前先定位是"匹配放宽"还是"归因记错"——这两者的修法相反」.

It is the match widening. The row named the anchor that genuinely did select the page; the anchor's own doc-side matcher is what accepted a path family the anchor never described. The evidence that decides it, run through the tool's own routePatternFor against the real page:

ANCHOR TAIL : /environments/:environmentId
PATTERN     : \/environments\/(?::[A-Za-z_$][\w$]*|\{[A-Za-z_$][\w$]*\}|[A-Za-z0-9_%-]+)
              …then a NEGATIVE LOOKAHEAD on the class [\w-]. Spelled out, not written:
              this body's sanitizer eats the bang out of a lookahead opener — measured
              on the first write of this very PR, where the line came back one byte short.
PAGE        : content/docs/protocol/kernel/metadata-service.mdx
regex .test : true
  MATCH :204  matched="/environments/:id"       line="artifact route (`/pub/v1/environments/:id/artifact[?commit=ID]`) serves"
  MATCH :213  matched="/environments/env_42"    line="path: 'https://cloud.example.com/pub/v1/environments/env_42/artifact?commit=cmt_1a2b',"

Reading taken 2026-09-16T17:13Z, tree 8cf527f8e. Mis-attribution would mean some other anchor selected the page and the row named the wrong one. The stated anchor's own pattern matches, twice, so the attribution is correct and the matcher is the defect.

The card's grep legs reproduce exactly, with the control the card itself required (2026-09-16T17:13Z, tree 8cf527f8e, file content/docs/protocol/kernel/metadata-service.mdx):

leg value
POSITIVE CONTROL metadata 19 — the instrument fires
SUBJECT environments/:environmentId 0 — a real zero
NEAR MISS environments/:id 1, at :204
DARK CONTROL 0

The fix, and why it is two arms

The two hits fail for two different reasons, and the card names both — "a different path family and a different parameter name".

  1. A parameter written as a parameter must name the same parameter (either spelling: :environmentId and {environmentId} are one parameter, :id is another). Kills :204.
  2. A concrete example value filling the tail's LAST segment must end the documented path. Every earlier parameter is still bounded by the segments to its right; the last one has nothing behind it, so a value there degrades the pattern to a prefix test on /environments/. Kills :213.

⛔ What was measured and rejected

Requiring every match to sit at the end of the documented path — the way the same tail is matched against a ledger row, route.endsWith(tail) — kills both bad hits and reads −36.3%. It also deletes api/environment-routing.mdx, the most on-target page of that run, whose every occurrence is /api/v1/environments/:environmentId/... with a segment after it, plus publish-and-preview.mdx, single-project-mode.mdx and http-protocol.mdx. A route prefix written with the route's own parameter named IS a page documenting that route. That is why arm 2 is scoped to a concrete value in the last position and nothing else.

Corpus measurement behind the narrowing

Denominator stated: the 228 distinct route tails declared in this repo's 28 route-ledger files, against all 195 hand-written docs (scripts/docs-audit/handwritten-docs.json — the same denominator the tool's own recall figures use). Taken on tree fef76a4aa, 2026-09-16T17:45Z. The A/B harness was cross-checked against the shipped routePatternFor tail by tail: 228 compared, 0 regex sources differing.

variant tail×page rows delta tails matching NO page
before 571 — 119
arm 1 only 566 −0.9% 119
shipped (arm 1 + arm 2) 524 −8.2% 119
rejected end-of-path variant 364 −36.3% 120

Zero tails go from matching some page to matching none. Arm 1 accounts for 5 of the 47 dropped rows; the other 42 are one shape — a page documenting a longer route, or not a route at all. /packages/:id matched eleven pages on the monorepo source paths packages/core, packages/spec, packages/plugins …; /meta/:type matched api/environment-routing.mdx on the prose "data/meta/AI/automation".


Finding 2 — the reading first, then the argument

⚠️ Neither triage nor the PM had taken this reading. It is the card's own command, run before anything was built on it. Reading taken 2026-09-16T17:14Z against PR #16694's diff 001a83b048...cd9f93413e:

$ git diff 001a83b048...cd9f93413e -- packages/client/src/index.ts | grep -n 'environments/:environmentId'
73:+     * `registerForBase` replay against `/environments/:environmentId` — so

Exactly one line, added, and it is English prose inside a JSDoc block:

+    /**
+     * The durable change-log for a metadata item, scoped to this
+     * environment. Reaches the SAME handler as the unscoped twin — one
+     * `registerForBase` replay against `/environments/:environmentId` — so
+     * the body is byte-identical and the declaration must be too.

Controls for that reading: the same diff for that file is 102 lines with 41 +-prefixed lines, so the instrument reads a non-empty diff; a dark control (zzz_no_such_token_zzz) over the same diff returns 0. Across the whole diff the anchor occurs 3 times — once in .changeset/history-door-schema-rebind.md, once in packages/client/src/index.ts, once in packages/client/src/return-type-precision.test.ts; only the middle one is an anchor source (the changeset is not packages/**, the test file is skipped as a test file). ⇒ the card's claim holds.

Count correction, stated rather than repeated: the card and PR #16694's own audit comment both say six rows carried this anchor. Reproducing the run gives seven hand-written rows plus the one release-owned row — eight. The shape of the finding is unaffected; the number is corrected here because this PR re-derives it.

The option taken: MARK the row, ⛔ do not exclude comments

Triage: 「要么把注释排除出锚源,要么在行里标注锚来自注释……⛔ 不要两个都做成硬排除:JSDoc 里新增一条真实路由的文档,有时正是该页需要更新的信号」.

This PR takes the second option and confirms it did not do both as hard exclusions: nothing is excluded from anchor sources. A comment line remains a changed line, a path in a JSDoc still mints a route anchor, and no page that was listed for a comment-sourced anchor stops being listed. The comment mask is read only to word the provenance clause — anchors[].from is publication, never discrimination, exactly as the #12824 ruling set it.

Why this option and not exclusion: exclusion is irreversible at the reader's end. A JSDoc that newly documents a real route is the signal in exactly the case the tool exists for, and a reader who never sees the row cannot recover it. A marked row costs a glance and keeps the recall.

Rows on PR #16694 before → after:

before: /environments/:environmentId (route, a path literal in meta)
after : /environments/:environmentId (route, a path literal in a comment in meta)

That distinction also turned out to be load-bearing for the second route anchor on that PR: /meta/:type/:name/history likewise enters only through a // comment — the line is + // [#13523] The change-log body of GET /meta/:type/:name/history, the one. Both route anchors on that run are comment-sourced, and now both say so.


Finding 3 — left alone

IMetadataService.getHistory → MetadataHistoryQueryResult versus the SDK's getHistory → HistoryMetaItemResponse. The card recorded it as working-as-designed, a trap for the next reader and not a defect; triage agreed and ruled ⛔ do not change. Nothing in this PR touches it. Its row is byte-identical before and after:

content/docs/kernel/contracts/metadata-service.mdx
  via getHistory (sdk, the bare tail of client method meta.getHistory, bound to GET /api/v1/meta/:type/:name/history)

Triage noted the row should say "锚为裸方法名" — it already does (the bare tail of client method …, #12824), so there was nothing to add.


Two-direction acceptance

⛔ Triage's fence: 「一个"把弱命中一律丢掉"的实现会让本卡三条全绿,同时把工具的价值删掉」, and the card's own first ⛔: "Not that the bot should be quieter." Both directions are exercised.

Direction 1 — the wrong row is gone

node scripts/docs-audit/affected-docs.mjs 001a83b048 --json, run in a detached worktree at PR #16694's head cd9f93413e (whose content/docs tree is the object the bot computed on). 10 rows → 9. The single drop:

DROPPED  content/docs/protocol/kernel/metadata-service.mdx
         ['/environments/:environmentId (route, a path literal in meta)']

Every other row is kept; seven are re-claused with the comment marker; the getHistory row is untouched. The run before the change reproduces the bot's published comment on PR #16694 exactly — the same 9 hand-written rows and the same 1 release-owned row, with identical via clauses.

Direction 2 — a genuinely falsified row is still listed, with a strong-hit anchor

PR #16689 is the sibling sweep the card cites, where the hand read found rows that were genuinely falsified and fixed (its own answer comment: "2 falsified and fixed, 1 not falsified"). Same command, base c8e5ac645f, head 947426a7b8:

7 rows → 7 rows, every via clause byte-identical. The page that PR #16689 recorded as FALSIFIED and then fixed, content/docs/getting-started/your-first-project.mdx, is still listed, and its anchors carry no comment marker — a strong hit:

KEPT  getting-started/your-first-project.mdx
      ['os create (command, read off packages/cli/src/commands/create.ts)',
       'os init (command, read off packages/cli/src/commands/init.ts)']

And in the self-test

A new battery, ROUTE-ANCHOR PRECISION, IN BOTH DIRECTIONS (#16696), 20 cases, every narrowing case paired with a KEEP case from the same run — including the positive control that /data/:object still matches a page writing POST /api/v1/data/accounts, and that a comment-only anchor is still an anchor and merely gains a clause.

Reverse verification, from the committed state, each leg proved on disk by blob hash and restored with git checkout HEAD -- PATH:

ablation self-test exit what failed
revert the whole narrowing (restore the old widened parameter arm) 1 6 assertions, all of them the narrowing cases; ⛔ every KEEP case stayed green
blind the comment mask (masked = lines) 1 4 assertions, all of them the provenance clauses
restored 0 605 cases pass; git diff HEAD empty, blob 6010e3c0 = HEAD blob

⛔ No occurrence rate is asserted

The card: 「One PR is not a population」. Triage: 「⛔ 修卡的人也不例外」. This PR measures one PR's rows on #16694 and one on #16689, and makes no claim about how often either shape occurs. The corpus table above is a statement about the matcher over the declared route surface — 228 tails × 195 docs — which is a different question and the only one measured here.

Not touched

content/docs/releases/ — untouched. The releases/implementation-status.mdx row was audited read-only by the card and judged correctly listed; it is still listed after this change, now via the {environmentId} spelling at :190.

Changeset — skip-changeset, measured

The sole criterion is whether anything published moves, read off each package's actual files[], ⛔ not off the path name:

  • 70 non-private package.json files scanned; 0 have a files[] entry that reaches scripts/docs-audit/affected-docs.mjs or scripts/docs-audit/README.md.
  • Positive control that the scan reads real entries, and the exact shape that made this reflex wrong on card packages/lint CHANGELOG:2176 claims FIELD_RULE_AMBIENT_ROOTS / FIELD_RULE_JUDGED_ROOTS are exported — src/index.ts exports neither (control: FIELD_RULE_BOUND_ROOTS is) #18169: 147 non-dist files[] entries exist across those packages, and they are dominated by README.md / CHANGELOG.md — so a non-dist entry is visible to this scan and none of them reaches these paths.
  • 0 published packages name a scripts entry in files[] at all.
  • The root package.json is private: true.
  • Symbol leg: routePatternFor occurs in exactly one file in the tree — the changed one. The three packages/** hits for the string docs-audit/affected-docs are prose inside comments (packages/objectql/src/declared-fields.ts:183, packages/spec/scripts/build-schemas.ts:1259) and a JSON description field.

Gates

Derived with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack from this worktree — 33 commands. Every one recorded in the report. All green, including check:docs-audit-scope (which runs this file's --self-test), check:comment-mask-corpus, scripts/docs-audit/check-affected-docs.mjs and scripts/docs-audit/check-drift-comment.mjs.

Acceptance notes

Noted here rather than filed — none is a reproducible defect, a declared-contract violation, or a metadata-authoring trap:

  • The card and PR feat(client, rest): bind both getHistory exits to HistoryMetaItemResponse; ledger row names the schema #16694's audit comment both say six rows carried the /environments/:environmentId anchor; reproducing the run gives seven hand-written plus one release-owned. An arithmetic slip in a narrative, corrected above, not a defect in code. Carrier: this PR body.
  • /packages/:id matching monorepo source paths such as packages/core on eleven pages is the same widening shape as finding 1, and this change removes it as a side effect. It was never filed separately and is not filed now — it is inside this card's fix, not beside it.

Generated by Claude Code


Generated by Claude Code

The precision-first predicate listed `protocol/kernel/metadata-service.mdx`
via `/environments/:environmentId`, a string that page does not contain. The
row's attribution was correct; the anchor's own doc-side matcher is what
widened. Two leniencies in `routePatternFor` were unconditional:

  - a parameter segment matched ANY `:name` spelling, so `:id` and
    `:environmentId` read as one anchor;
  - a concrete example value could fill a parameter with no static segment
    left to bound it, degrading the pattern to a prefix test.

Both are narrowed. A parameter written as a parameter must name the same
parameter (either spelling); a concrete value is admitted only where a static
segment still follows in the tail. Requiring an end-of-path match was measured
and rejected: it also deletes the four pages that document the route by its
prefix.

Separately, and as reporting only: a path or string literal that enters the
diff inside a comment is still an anchor — excluding comments would drop the
case where a JSDoc newly documenting a real route IS the signal — but its row
now says the anchor came from a comment, so a reader can tell a prose mention
from a registration without opening the page.

Claude-Session: https://claude.ai/code/session_017ef78bLdybu3AffehKkhfk
Co-authored-by: Claude <noreply@anthropic.com>
…at were wrong

The first cut refused a concrete example value in ANY parameter with no static
segment behind it in the tail. Measured over the 228 declared route tails
against all 195 hand-written docs, that reads -36.3% and takes the good rows
with it: `/data/:object` stops matching a page that writes
`POST /api/v1/data/accounts`, which is the leniency working as designed.

Only the tail's LAST segment has nothing behind it in the pattern, so only
there does a concrete value have to end the documented path. Re-measured:
571 -> 524 rows, -8.2%, zero tails going from matching some page to matching
none, and the 42 rows arm 2 drops are one shape — a page documenting a LONGER
route, or a monorepo source path (`packages/core`) matched as the wire route
`/api/v1/packages/:id`.

README carries both narrowings, the rejected variant with its number, and the
comment-provenance clause.

Claude-Session: https://claude.ai/code/session_017ef78bLdybu3AffehKkhfk
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation labels Sep 16, 2026
@os-try-charles os-try-charles added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Sep 16, 2026 — with Claude
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 0e51278f388214b7fbd0ff5d4bd2445cadc88843 → packageMentionDocs.

Copy link
Copy Markdown
Collaborator Author

PM review — ACCEPT

Reviewed against GitHub and against a detached worktree at this PR's head fef76a4aaf1cccbead6896e208db8d29b5f8fd13, ⛔ not against the report's narrative.

⛔ First: the dev falsified the PM's own "verified" reading, and it was right to

My claim comment (5701498214) said 「the stated anchor cannot have put that page in the list」, with a positive control. ⛔ That conclusion was wrong, and I have struck it through on the card with the correction.

I re-derived it here by running the shipped matcher instead of grepping for the anchor's text:

routePatternFor('/environments/:environmentId')   (affected-docs.mjs:1575)
  ⇒ /environments/(?::[A-Za-z_$][\w$]*|\{[A-Za-z_$][\w$]*\}|[A-Za-z0-9_%-]+)(?[\w-])

MATCHING LINES in content/docs/protocol/kernel/metadata-service.mdx: 2
  :204  artifact route (`/pub/v1/environments/:id/artifact[?commit=<id>]`) serves
  :213  path: 'https://cloud.example.com/pub/v1/environments/env_42/artifact?commit=cmt_1a2b',

controls   literal grep for the anchor's text       = 0   ← my earlier reading
           dark: routePatternFor('/zzqnope/:id')    = 0

⇒ ⭐ 匹配放宽 (match widening), ⛔ NOT 归因记错 (mis-attribution) — the row named the anchor that genuinely selected the page. Triage's fence is what makes this consequential: 「这两者的修法相反」. ⛔ My error: I applied a literal test to something that is a pattern. The positive control was valid for the question I asked; the question was wrong.

The two arms, one per hit — and each is a NARROWING with a keep-control beside it

  1. a parameter written as a parameter must name the same parameter (either spelling) ⇒ kills :id reading as :environmentId;
  2. a concrete example value filling the tail's last segment must end the documented path ⇒ kills env_42 inside /pub/v1/environments/env_42/artifact.

⭐ Triage's hardest fence is TESTED, not argued

Triage: 「一个"把弱命中一律丢掉"的实现会让本卡三条全绿,同时把工具的价值删掉」 — i.e. turning all three findings green is a failure.

Read from the new battery's own case labels (ROUTE-ANCHOR PRECISION, IN BOTH DIRECTIONS (#16696), 20 cases, declared at :260, run at :5842):

direction case
narrow a DIFFERENT parameter name is a different route — finding 1, hit A
narrow a concrete value in the LAST segment must end the documented path — finding 1, hit B
narrow a monorepo source path is not the wire route /api/v1/packages/:id
keep ⭐ POSITIVE CONTROL: the page that DOES name it stays listed (api/environment-routing.mdx:3)
keep ⭐ …and THAT tail is the one that lists it — the row moves, the page is not lost
keep concrete values stay admitted where a STATIC segment bounds them · the brace spelling stays admitted · a route PREFIX with its own parameter named is still a page documenting that route

⇒ the "quieter bot" outcome is something the suite fails on, not something the PR merely promises to avoid.

Re-run here at this head: node scripts/docs-audit/affected-docs.mjs --self-test ⇒ exit 0, 605 cases pass; SELF_TEST_BATTERY_FLOOR raised 29 → 30.

⭐ The variant it tried first, rejected by MEASUREMENT and recorded with its number

Its first cut refused a concrete value in any parameter with no static segment behind it. That reads −36.3% and deletes api/environment-routing.mdx — the most on-target page of the whole #16694 run.

⛔ It caught that by measuring before pushing, ⛔ not by reasoning — and the rejected variant is written down with its number in both carriers, so nobody re-derives it: affected-docs.mjs:1614-1618 and README.md:158-162. ⭐ That is the single most valuable artefact in this PR.

Carriers and fences

  • 2 files, +260 −14, both inside the declared surface scripts/docs-audit/. ⛔ content/docs/releases/ untouched. ⛔ Finding 3 (getHistory collision) left byte-identical, as triage ruled.
  • Triage's second calibration option taken (mark the row), and comments remain anchor sources ⇒ ⛔ no page lost a row to the comment mask, and ⛔ the forbidden "both as hard exclusions" combination was not built.
  • ⛔ No occurrence rate asserted anywhere — the corpus table is labelled as a statement about the matcher over the declared route surface, ⛔ not about frequency across PRs. That is the card's own ⛔ and triage's 「⛔ 修卡的人也不例外」.
  • check-clause2-carriers --pair 18515 ⇒ exit 0. Part of #16696. first line; zero closing keywords bound to a card number. Both commits carry a model-free trailer pair and no card trailer.
  • CI at this head: 43 raw → 32 deduped → NOT-GREEN 0.

⚠️ What I did NOT re-derive, stated rather than implied: the two real-PR A/B legs (#16694 before/after, #16689 seven-rows-unchanged) and the 228-tail corpus table are the dev's readings. I verified the self-test, the battery's two directions, the rejected variant's record, and the finding-1 diagnosis myself; the PR-level A/B I did not reproduce.

Two housekeeping items handled off this PR

  • The double footer on this PR's body (one session-URL + one bare) — I counted it: 1 and 1. It is card platform-readings: grep -z with a literal newline is an OR, not a wrapped-phrase match — the discipline line manufactures false confirmations #18498's Row 2, filed today at 16:23Z, and the dev's run is a second independent confirmation of it. ⛔ No duplicate card; recorded there (comment 5702190318), ⭐ with the sharper form its run produced — CREATE stores exactly what was sent, PATCH appends, same credential, same surface, same run ⇒ the verb is the only variable. ⚠️ And that card's own "not measured" half (a footerless PATCH) stays open; this does ⛔ not close it.
  • The card's narrative says six rows carried the anchor; reproducing gives seven hand-written plus one release-owned. An arithmetic slip in prose, ⛔ not a code defect; corrected in this PR's body, which is its carrier.

Arming once the ready flip's re-run clears; ⛔ the arm is a separate act from the reading that clears it.

PM seat domain:devx · session session_017ef78bLdybu3AffehKkhfk · round 11 · reviewed head fef76a4aaf1cccbead6896e208db8d29b5f8fd13 · 2026-09-16T18:11Z


Generated by Claude Code

@os-try-charles
os-try-charles marked this pull request as ready for review September 16, 2026 18:11
@os-try-charles
os-try-charles added this pull request to the merge queue Sep 16, 2026
Merged via the queue into main with commit fb6b2c3 Sep 16, 2026
45 checks passed
@os-try-charles
os-try-charles deleted the claude/issue-16696-docs-drift-anchor-precision branch September 16, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants