feat(translate): translate in a checked loop instead of one call - #350
Open
youyongsong wants to merge 11 commits into
Open
feat(translate): translate in a checked loop instead of one call#350youyongsong wants to merge 11 commits into
youyongsong wants to merge 11 commits into
Conversation
…round trip Link targets, code (no length floor), JSX attribute values, raw-HTML href/src, heading anchors, bare URLs, MDX expressions and reference labels are replaced by opaque placeholders before the document reaches the model, then restored. The model never holds the real value, so it cannot rewrite it — which the previous approach, a detailed "CRITICAL: do not modify ANY link content" prompt section, demonstrably failed to prevent. Restoration is also the gate: a placeholder that comes back fewer times, more times, in a node kind it was not issued for, or one that was never issued at all, raises MaskIntegrityError naming the file and the placeholder. There is no lenient path — shipping a damaged translation is how the damage stayed invisible. Which attributes carry prose is declared once, beside the components, in runtime/components/translation-policy.ts; everything else masks by default so an unclassified component fails towards a visible untranslated label rather than a silent identifier rewrite. Supersedes replaceCodeBlocksWithPlaceholders/restoreCodeBlockPlaceholders, whose 50-character floor left short blocks exposed. Verified on 1789 real documents (914 en sources, 875 zh translations): 1788 restore byte-identical; the one difference is a pre-existing non-idempotency in remark's stringifier that reproduces with no masking at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g element `detectGaps` treated any node carrying a `children` key as covered by walking its children. A self-closing MDX element is a parent with an empty `children` array, so its end offset was never recorded: when it was the last node in a file, everything from its start to EOF became a gap and every message inside it was filtered away. A canonical API reference page — a heading and a single `<K8sAPI … />` — is precisely that shape, so the rules aimed at those pages reported nothing at all while the build stayed green. The rule specs never caught it because none of them ran message control. A `lintMdxPipeline` helper now does, and the new cases fail against the unpatched dependency (3 of 4) and pass with it. Fixing it surfaced what it had been hiding: `no-unresolved-api-ref` read `metadata.name` off each permission source, while the permission plugin builds its runtime module from `items`. Against a `kind: List` source — this repo's own fixture included — it knew no function names and called every `<K8sPermissionTable>` reference unresolved. Both shapes are now read the way the runtime reads them, and the permission example, which referenced two FunctionResources that do not exist in the fixture, points at two that do. The dependency change is carried as a patch-package patch and belongs upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing compared the two before. A translation could drop a bullet, rewrite a link into a different page that happens to exist, or come back verbatim in English, and every gate stayed green — the build only ever notices a link that resolves nowhere, and only after the damaged file has been committed back. The `translation-parity` rules read two documents, paired exactly by the `sourceSHA` the translator writes. `translation-up-to-date` establishes the pairing and the rest stand down without it, out loud rather than in silence. Links are compared by resolving each one against the document that holds it and dropping the language segment, which is what makes source and translation comparable at all: assets legitimately point back into the source tree because they are not copied per language. `doom translate check` runs the same rules over existing translations offline, with no model involved — the way to survey ~40 repositories before deciding what is worth re-translating. Measured over 1768 real translated documents: 14 findings, every one verified by hand, none false. Among them three documents whose prose came back in English with only frontmatter translated, a `<Term>` wrapped in backticks so it renders as literal source, two hyperlinks dropped from a page, and an English source whose frontmatter is missing its opening `---`. Two rules were narrowed to get there and both cases are recorded where the narrowing lives: comparing `src`/`href` as written reported every illustrated page, and matching anything shaped like `name.tld/path` reported API groups. `translation-terminology-adherence` ships unregistered — against the shared terminology table it reports 490 problems on a corpus that is substantially correct, because that table is a glossary of preferences, not invariants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`doom lint` reported success over a directory of translations having applied no rule to it. The remark config — which carries every doom lint rule, dead links included — was attached only to files under the source-language directory, so `zh/` and `ru/` documents were parsed and then checked against nothing. Linting a translation and linting nothing produced identical output, which is why the proposal that motivated this work concluded translations were unreachable by lint for a different reason (ordering) and never noticed this one. It now covers every language directory, still scoped to language directories rather than to every markdown file in the repository: changesets and design notes are not documentation pages. Spell checking stays on the source language, where its dictionaries belong. The `translation-parity` rules also move ahead of `check-dead-links`, which rewrites link urls in place as a side effect of resolving them; comparing a rewritten translation against an unrewritten source reported 961 problems on a corpus that has one. Fallout in this repository, all of it previously invisible: `fixture-docs` is now ignored outright — it is test material whose fixtures deliberately reference resources that do not exist, and its language layout is the mirror image of `docs/` — and `docs/en/usage/permission.mdx` is re-paired with the source it belongs to. Two translations, `docs/en/usage/api.md` and `docs/en/usage/configuration.md`, are genuinely stale against their Chinese sources and are left reported: clearing them needs a translator run, which is the point of the rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`remark-lint-match-punctuation` and `remark-lint-no-chinese-punctuation-in-number` pair and place CJK punctuation. They joined the rule set when the only documents anyone linted were Chinese; now that translations are linted too they read `pod’ами` — which is how Russian declines a Latin word — as an unmatched quotation mark, and report every page that does it. On a real corpus of 1728 translated documents that was 17 of 19 findings, none of them a defect. Both rules now apply only under a `zh` directory. A document whose language cannot be told from its path keeps its messages: not knowing is a reason to report, not a reason to go quiet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: fda6f77 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
youyongsong
marked this pull request as draft
August 25, 2026 15:36
added 6 commits
August 25, 2026 17:13
`doom translate check` read a file and linted it. The translator needs the same judgement about a document that is not on disk yet, and the harness needs it again to decide whether the agent may finish. Three callers, so the loop body moves out into `createTranslationChecker`, and "the agent is held to the rules it checks itself with" stops being a convention. The rule list moves with it, into `remark-lint/rule-sets.ts`. The translator gets the whole lint pipeline minus the three rules that read state outside the document — `check-dead-links` scans a docs tree the translator is in the middle of writing, and reports the pages it has not reached yet. Deriving that set by subtraction rather than by listing means a rule added to doom guards machine-translated documents the same day. Also drops the anchor caches for a document before checking it: both key on a path and assume a path names one fixed document, which stops being true when the same target is checked once per repair turn.
Masking only holds if the unmasked bytes are not reachable. They are not written into the scratch directory, which is most of it — but pi's `NodeExecutionEnv` takes a `cwd` that only resolves *relative* paths, so an absolute path goes wherever it says. An agent that read the real source that way would produce a perfectly green run, which is the failure mode this work exists to remove. So the environment the tools run against is wrapped: outside the scratch, every path answers `not_found` — the same answer as a path that does not exist, so the boundary is not also an oracle. Symlinks are resolved before the check, temporary files are redirected inside, and `exec` is refused outright: the agent is given no shell tool, and the reason there is no shell does not stop being true if someone later adds a tool that uses one. Each containment case in the spec is paired with the same call against an unjailed environment, so the test says what the agent would otherwise have reached. Removing the confinement fails 6 of the 10 cases; the other 4 do not test containment.
A translation was one `chat.completions.create` with the whole file in it and nothing looking at what came back. That is where every corruption in this proposal's record comes from: rewritten links, collapsed path segments, a bullet that simply disappeared. Masking took the link targets out of the model's reach; this takes the *result* out of its hands. `doom translate` is now orchestration and a gate. It still decides what gets translated — globs, `sourceSHA`, copy-only directories, frontmatter merging, the deterministic rewrites — and hands each document to a loop that ends when the document passes its checks. The loop is pi's, and the load-bearing part is which of pi's hooks the check goes in. `getFollowUpMessages` fires when the agent has run out of tool calls and would stop; whatever it returns puts the agent back to work. So the check is the exit condition rather than something that runs afterwards: the agent has no name for it, cannot call it, and cannot finish while it still reports anything. The turn cap sits in `shouldStopAfterTurn`, which pi consults first, so a run that will not converge ends — and ends failed. There is no third exit where a document ships anyway, and there is no flag that turns any of this off: the old single-call path is gone rather than kept as a fallback. Failures are collected and raised at the end, with the whole list. A red pipeline is the only channel in this organisation that has ever reached anyone; every one of the seven manual repairs on record was triggered by one. Notes on the pieces: - Context is released deterministically — the bodies of older `read` results, which are still on disk in the scratch directory and can be read again. pi's own compaction summarises the conversation with an LLM, and the conversation is full of bytes that have to survive exactly; only its *measurement* is borrowed, to decide when to release. Wording the document has already used is carried across a release as a fact, not a glossary. - pi's `read` tells the model to fall back to `sed` when a line is too long. There is no shell here, so that sentence is rewritten to point at a `read_bytes` tool. The longest line in the current corpus is 2.7KB against a 50KB limit — which is a reading of today's corpus, not a reason to leave the escape hatch dead. - `check` returns findings and never the restored document: handing that back would put the link targets masking removed straight back into the context. - pi is loaded lazily. Importing it costs ~450ms and adds ~89MB to `node_modules`, and doom is `yarn add`ed by ~40 repositories on every build, so `doom build` and `doom lint` must not pay for it. - Both pi packages are pinned exactly. They are pre-1.0 and this is a shared build tool. Removing `getFollowUpMessages` fails 4 of the 9 loop cases.
`**bold**` only works when the delimiters sit against the text. `** bold **` does not, and neither does `**注意：**卸载` — the closing run is preceded by punctuation and followed by a letter, so CommonMark leaves it as characters. The reader sees the asterisks. Nothing else notices: the document parses, every link resolves, every component is accounted for. Found by translating for real. `gpt-5.6` produced `**重要提示：**卸载` from a source that read `**Important:** Before`, and the translation passed every check there was. Measured before turning on: 2 of 922 English documents, 7 of 875 Chinese, 2 of 875 Russian — 11 in all, and every one is a page with literal asterisks on it. Two of the English ones are the source of the Russian pair. `__` is covered by the same rule and its tests but does not occur in this corpus, so it rests on the spec rather than on a measurement, and the comment says so. Also moves `no-unmatched-anchor` out of what the translator checks. It failed `zh/install/installing.mdx` on an anchor in `install/prepare/download.mdx` — a different document, whose stale translation had lost the heading carrying it. The link and its anchor are both masked, so they came back byte-identical to the source; nothing about the document being translated was wrong and no repair round could have changed the outcome. It joins the rules that judge a document by the state of its neighbours, and keeps running in `doom lint` over a tree that is complete.
Every deterministic check in this pipeline asks a structural question, and
all of them pass a translation that is well-formed, correctly linked and
about something else. `state.md` §7 records two documents that lost whole
sentences and were caught only because the missing sentence happened to
contain a `<Term>`. So a second reading, by a model, comparing both documents
and saying what the translation lost, added or got wrong.
Three things keep that from being a coin toss: only omission, addition and
mistranslation block — readability is reported and never fails a build; a
finding must be drawn twice, so a false positive has to happen twice to reach
anyone; and every finding quotes the passage it is about, so it can be
checked instead of believed.
It reads the *masked* documents, like the translator does. Its findings go
back into the translator's context, and an unmasked link target quoted in a
finding would be a link target back within the model's reach.
Calibrated before it was allowed to block, because the ruling is that it
blocks from day one and there is no report-only period. Two numbers, on
acp-docs, `gpt-5.6` at medium:
- false reds on translations believed good: 0 of 30. Legible rather than
mysterious: 7 of the 60 readings did say something (11 findings), and
agreement removed all of it.
- injected recall — a block deleted from a good translation: 18 of 20 named
the deleted block, 1 flagged something else, 1 said nothing.
Getting there took two corrections, both measured:
- The judge was shown the frontmatter, which the translator is *supposed* to
change, and dutifully reported it as an addition on every document.
- It was not shown the glossary the translator gets, so it reported
"workload cluster" → "业务集群" — what these documents call it — as a
mistranslation. Four of the ten first-round false reds were that one term.
`resolveTerms` now has one implementation, used by both.
With those fixed, false reds went 13.3% → 0% and recall 17/20 → 18/20. On a
held-out sample of 40 unseen pairs the rate is 7.5%, and three of those four
findings are arguable rather than wrong — one of them found a contradiction
in the *English source*, where a table says "when the cluster type is
Standard" twice with opposite outcomes.
Also, from translating for real: the gateway refuses requests under load
("429 requests-per-minute", "Upstream service temporarily unavailable"), and
pi surfaces that as a stream error that ends the document. A corpus is
thousands of calls, so meeting one is routine and failing the document over
it fails the wrong thing. Both paths now retry with backoff; the translator
resumes from the draft already on disk rather than starting over. Observed
working on a live run: one document was refused, retried, and finished.
A document that starts
weight: 13
---
has no frontmatter. What it has is a heading: a line of text underlined by
`---` is a level-two heading, so the page gets one called "weight: 13" and
loses whatever the frontmatter was going to say.
Nothing noticed. The file parses, links resolve, `heading-increment` sees a
rank drop from 2 to 1 and is content, `title-required` finds the `#` heading
below. It took translating the document to surface it — comparing the two
showed a heading the translation did not have — and by then the finding lands
on the translation pipeline, where nobody can act on it: the repair is in
another repository, and translating "weight: 13" as a heading would only make
the translation worse. Detecting it here puts it where it can be fixed.
Measured on acp-docs before turning it on: 914 English documents, 88 of them
legitimately without frontmatter (generated api pages, index pages), and
exactly one of those 88 opening with something shaped like a frontmatter key.
One true positive, no false ones. "Every document must have frontmatter"
would have reported all 88.
The key list is counted rather than remembered — weight 797, sourceSHA 194,
title 132, i18n 125, queries 73, description 21, category 5, author 4 — and
deliberately a list rather than a general pattern, because a heading really
can begin "Note:" or "Step 1:".
youyongsong
changed the title
youyongsong
marked this pull request as ready for review
August 26, 2026 05:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Translated documents came back damaged and nothing looked at them. This replaces the translator: a single model call with the whole file in it becomes a loop that ends when the document passes its checks.
What was actually happening
Six kinds of corruption were measured on the real corpus over 2026-08:
<ExternalSiteLink … />flattened into a markdown link,../virtualization/virtual_machine/collapsed into../virtualization_virtual_machine/,../global_dr.mdxrewritten as../global.dr.mdx,how_to/rewritten ashow-to/, the same link generated twice, and whole sentences disappearing.Only the corruptions that happen to produce a link resolving nowhere are ever noticed, and only at the end, by a build failure — after
upload-translationshas already committed the damaged file back. A page that lost a sentence builds green forever.The prompt already carried a detailed
**CRITICAL**: Do not translate or modify ANY link contentsection listing every link form. It did not hold. Two documents lost whole sentences with the build green throughout: a Chinese page dropped "if you do not have Customer Portal access, contact support", and a Russian one dropped the product name from the opening line of the installation guide.Four layers
1. The model no longer sees what it must not author
cli/translate-mask.ts. Link targets, image sources, code (fenced and inline, no length floor), MDX JSX attribute values,href/srcin raw HTML, custom heading anchors, bare URLs, MDX expressions and reference labels are replaced with opaque placeholders before the document is sent, and restored afterwards. The real value is not in the model's context, so it cannot rewrite it.Restoring is also a gate.
MaskIntegrityErrornames the file and each offending placeholder when one comes back fewer times than it went out (a node was deleted), more times (duplicated), was never issued (invented), landed in a node kind it was not issued for, or when the response does not parse. There is no lenient path.Which JSX attributes carry prose is declared once, in
runtime/components/translation-policy.ts. Everything else is masked, so a component nobody has classified yet fails towards a visible untranslated label rather than a silent identifier rewrite.Masking only ever touches AST nodes and never pattern-matches inside prose, so it cannot swallow content that should have been translated. Verified over 1789 real documents: mask → restore reproduces the document byte-for-byte in 1788, the one difference being a pre-existing non-idempotency in remark's own stringifier that reproduces with no masking at all.
2. Checks that read two documents
remark-lint/translation-parity/*, paired exactly by thesourceSHAthe translator writes:translation-up-to-date(the pairing itself — and every other rule stands down without it, out loud rather than in silence),link-isomorphism,component-multiset,jsx-attribute-parity,echoed-source,heading-sequence,frontmatter-preservation,length-ratio,url-residue.doom translate check [root]runs them over existing translations with no model involved — the offline way to survey which documents are damaged.Measured over 1768 real translated documents: 14 findings, each verified by hand, none false. Among them three documents whose prose came back in English with only the frontmatter translated, a
<Term>wrapped in backticks so the page rendered its source, and two hyperlinks dropped from a page.3. A repair loop, with the check as its exit condition
cli/translate-agent.ts.translate.tskeeps the orchestration — which files,sourceSHAincrementality, copy-only directories, frontmatter merging, the deterministic rewrites — and hands each document to a loop.The loop is pi's, and the load-bearing part is which of pi's hooks the check goes in.
getFollowUpMessagesfires when the agent has run out of tool calls and would stop; whatever it returns puts the agent back to work. So the check is the exit condition rather than something that runs afterwards: the agent has no name for it, cannot call it, and cannot finish while it still reports anything. The turn cap sits inshouldStopAfterTurn, which pi consults first, so a run that will not converge ends — and ends failed.Built on
runAgentLooprather than theAgentclass, becauseAgenthardwiresgetFollowUpMessagesto an internal queue and does not expose it. Same reasonAgentHarnessis not used: it is a facade, and in 0.84.3 an unimplemented one.The agent works in a scratch directory it cannot leave.
NodeExecutionEnvtakes acwd, butcwdonly resolves relative paths — an absolute path goes wherever it says, and an agent that read the real source that way would produce a perfectly green run. Everything outside answersnot_found, symlinks are resolved before the check, and there is no shell.Context is released deterministically: the bodies of older
readresults, which are still on disk and can be read again. pi's own compaction summarises the conversation with an LLM, and the conversation is full of bytes that have to survive exactly.There is no flag that turns any of this off. The single-call path is deleted rather than kept as a fallback.
4. A semantic review
cli/translate-judge.ts. Every check above asks a structural question, and all of them pass a translation that is well-formed, correctly linked, and about something else — which is exactly how the two lost sentences above survived. A second reading compares both documents and says what the translation lost, added, or got wrong.Three things keep that from being a coin toss: only omission, addition and mistranslation block (readability is reported and never fails a build); a finding must be drawn twice, so a false positive has to happen twice to reach anyone; and every finding quotes the passage it is about. It reads the masked documents, because its findings go back into the translator's context.
Calibrated before it was allowed to block, on acp-docs with
gpt-5.6at medium reasoning:Getting there took two corrections, both measured: the judge was being shown the frontmatter, which the translator is supposed to change, and dutifully reported it as an addition on every document; and it was not being shown the glossary the translator gets, so it reported the house translation of a product term as a mistranslation — four of the ten first-round false reds were one term.
When it fails
Any document that cannot be made to pass fails the whole run, with the complete list, after every other document has been attempted. Nothing is uploaded. There is no per-file downgrade and no falling back to the previous translation: a red pipeline is the only channel in this organisation that has ever actually reached anyone — every one of the seven manual repairs on record was triggered by one — and a document that is quietly left at its old version is a problem nobody sees.
The escape hatch for one document is unchanged and belongs to a person:
i18n.disableAutoTranslationin its own frontmatter.Two new lint rules, each measured before being turned on
no-unparsed-emphasis—**bold**only works when the delimiters sit against the text.** bold **does not, and neither does**注意：**卸载, and the reader sees the asterisks. Found becausegpt-5.6produced exactly that and every check was green. 2 of 922 English documents, 7 of 875 Chinese, 2 of 875 Russian; every one a real page with literal asterisks on it.no-unopened-frontmatter— a document openingweight: 13/---has no frontmatter; it has a heading called "weight: 13". 914 English documents, 88 of them legitimately without frontmatter, and exactly one of those opening with something shaped like a frontmatter key.Both are in
remark-lint/rule-sets.ts, from which the translator's rule set is derived by subtraction — so a rule added to doom starts guarding machine-translated documents the same day.Verification
tsc0 ·eslint0 · 503 tests · type-coverage 100%.getFollowUpMessagesfails 4 of the loop's 9.gpt-5.6, including a 33KB document: every document passed, and the historic "whole sentence disappeared" defect is gone — the bullet is back and<Term>counts 5 → 5 where it used to be 5 → 4.node_modules+89MB and ~450ms to import, so it is loaded lazily inside the translate subcommand anddoom build/lint/exportpay nothing. Both pi packages are pinned exactly; they are pre-1.0 and this is a shared build tool.Before merging
install-dependenciesinstalls the latest published@alauda/doomon every build, so publishing is the release gate and a rollback is a released version, not a runtime switch.Once this ships, the two new rules will report on every repository's own content. On acp-docs they report 5 findings, all real, all fixed in
alauda/acp-docs#1064— the other repositories should each rundoom lintonce to see what surfaces.Design and evidence:
doc-centerproposals/translation-rework/.