Skip to content

Commit b9be867

Browse files
claude[bot]claude
andauthored
Survey: which scripts/** self-tests cannot prove they ran — and the two that now can (#13797)
* fix(gates): battery roster floor + verdict handshake for check-self-test-wired Its --self-test decided success by `failures.length === 0` alone and its dispatch discarded the call's completion, so an early return printed nothing and exited 0 -- in the gate whose subject is that a shipped self-test runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC * fix(gates): battery roster floor + verdict handshake for check-self-test-workflow-commands Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC * feat(scripts): measure-self-test-floor — census of self-tests that cannot prove they ran Two orthogonal holes, counted separately and never summed: no assertion floor (static, published criterion) and no verdict handshake (measured by injecting an early return and reading the exit code). Both instruments are driven against known-holed and known-sound fixtures on every invocation, and a control failure refuses instead of printing a smaller number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC * fix(gates): refuse the ledger-growth remedy in check-self-test-wired, and record the shape check-ratchet-remedy-authority went red on this branch and is green on main: the remedy sentence is older than this PR, but a comment added here named SELF_TEST_RUN_OTHERWISE and its shrink-only nature one line apart, which pulled the target inside anchorFor's 400-character window for the first time. The message now says what the ledger's own docblock already said -- wiring the self-test is the whole remedy -- and the legitimate row is named as a maintainer's judgement of evidence. Shape recorded in the control corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent f532630 commit b9be867

4 files changed

Lines changed: 730 additions & 10 deletions

File tree

scripts/check-ratchet-remedy-authority.mjs

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -701,6 +701,40 @@ const CONTROL = {
701701
why: 'PR for #11341. Refuses by binding a negation to the verb, over a shrink-only baseline holding only the child-environment leaks that predate the gate; their repair is a card per neighbourhood, so no authority is missing.',
702702
},
703703

704+
// The sixth refusal instance, and the first to arrive because a gate's own
705+
// SHRINK TESTIMONY MOVED rather than because its remedy did (#13489). Its
706+
// remedy sentence is unchanged from before that card and was reachable the
707+
// whole time; what changed is proximity. `anchorFor` reads the ±400 characters
708+
// around every mention of a DECLARED target, and this gate's `⛔ SHRINK-ONLY`
709+
// banner sits at the top of a docblock long enough to push the const's own
710+
// name out of that window. A comment added elsewhere in the file named
711+
// SELF_TEST_RUN_OTHERWISE and its shrink-only nature one line apart, stage 2
712+
// reached the target for the first time, and a remedy that had been offering
713+
// registry growth for as long as it existed became visible.
714+
//
715+
// ⛔ The lesson is about this detector, not about that comment: a gate whose
716+
// testimony sits just outside the window is INDISTINGUISHABLE here from one
717+
// with no ratchet at all, and the difference is a docblock's length. Recorded
718+
// rather than engineered around — the window is a deliberate precision choice
719+
// (widening it was measured and rejected), so the honest response is to know
720+
// that `excluded` can mean "not reached yet".
721+
//
722+
// Refusal — not marking — because the ledger's own docblock already says it:
723+
// "Nothing joins this list to silence a finding: a gate that really does ship
724+
// an unrun self-test gets wired instead." The author-facing message had been
725+
// contradicting that sentence; it now agrees with it. The legitimate row still
726+
// exists and is named, as a maintainer's judgement of evidence rather than as
727+
// an escape from wiring. Both refusal limbs fire on the rewritten sentence:
728+
// the negation bound to the verb, and the act denied as its own subject.
729+
//
730+
// ⚠️ The two shapes are quoted HERE and not in the `why` below. Putting them
731+
// in the string made this file read as an instance of its own convention and
732+
// reddened self-test case (1) — measured, on the first attempt at this entry.
733+
'check-self-test-wired.mjs': {
734+
expect: 'refused',
735+
why: 'PR for #13489. Refuses by both limbs at once over a shrink-only ledger — see the sentence quoted in the comment above — because wiring the self-test is the whole remedy and it is the landing author\'s. Reached for the first time when a comment elsewhere in that file put the ledger\'s name and its shrink-only nature one line apart, inside anchorFor\'s window; the remedy sentence itself is older than this card.',
736+
},
737+
704738
// The tenth, and the fourth to join as a NEW gate rather than by retrofit
705739
// (#13582). Marking rather than refusal, on the same reading as the seventh
706740
// through ninth: there IS a legitimate act here and it has an owner. The gate

scripts/check-self-test-wired.mjs

Lines changed: 157 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,12 @@ import { maskComments } from './js-comment-mask.mjs';
107107
const ROOT = join(fileURLToPath(new URL('.', import.meta.url)), '..');
108108
const WORKFLOW_DIR = '.github/workflows';
109109

110+
// The token every gate in this farm writes when it names a path belonging to a
111+
// maintainer rather than to the landing author (#8435). Declared per gate by
112+
// convention, and read out of AUTHOR-FACING strings — a comment mentioning it
113+
// tells an author nothing, so it has to live in the message itself.
114+
const RATCHET_AUTHORITY_MARKER = '⛔ MAINTAINER-ONLY';
115+
110116
/** Extensions whose files can be a `scripts/` entry point. */
111117
const SCRIPT_EXT = /\.(mjs|mts|js|sh)$/;
112118

@@ -263,10 +269,13 @@ export function auditPopulation({ carriers, named, selfTested, ledger }) {
263269
' `--self-test`, but no workflow ever executes it with that flag. An unrun\n' +
264270
' self-test is a phantom check, and for a gate whose defect class is its\n' +
265271
' matching rule it is also the ONLY instrument the rule has (#11150).\n' +
266-
` Wire \`node ${script} --self-test\` into the step that runs it. If the\n` +
267-
' self-test is already run some other way — the script drives another tool\'s,\n' +
268-
' or its ordinary run executes its own cases — add a SELF_TEST_RUN_OTHERWISE\n' +
269-
' row in scripts/check-self-test-wired.mjs naming the evidence for that.',
272+
` Wire \`node ${script} --self-test\` into the step that runs it. That is the\n` +
273+
' whole remedy, and it is the landing author\'s.\n' +
274+
' ⛔ Do not add a SELF_TEST_RUN_OTHERWISE row to clear this. That ledger is\n' +
275+
' shrink-only; adding an entry is not the fix, it is this finding written down\n' +
276+
' somewhere quieter. A row records that a self-test genuinely IS run another\n' +
277+
' way — a wrapper drives it, or the ordinary run executes its own cases — and\n' +
278+
` judging that evidence is ${RATCHET_AUTHORITY_MARKER}, never a way out of wiring.`,
270279
});
271280
}
272281
return findings;
@@ -410,14 +419,89 @@ function main() {
410419
// --self-test
411420
// ---------------------------------------------------------------------------
412421

422+
// ---------------------------------------------------------------------------
423+
// The self-test's own battery registry, floor and verdict (#13489)
424+
// ---------------------------------------------------------------------------
425+
//
426+
// `failures.length === 0` used to be this self-test's ONLY success condition,
427+
// so "every case held" and "the cases never ran" printed the same line. And
428+
// the dispatch below was `if (--self-test) selfTest()`, which discards the
429+
// call's completion: an early `return` anywhere above the verdict printed
430+
// NOTHING and still exited 0. Measured on 597020aa5 by injecting `return;` as
431+
// the first statement of `selfTest()` -- exit 0, zero bytes of output, in the
432+
// gate whose whole subject is that a shipped self-test actually runs.
433+
//
434+
// Both holes are closed the way PR #13487 validated on check-doc-authoring:
435+
// what is pinned is the registered NAMES, not a number. Every section opens
436+
// with `battery('<name>')`, every `ok()` is attributed to the battery most
437+
// recently opened, and the floor requires the OPENED set to equal the DECLARED
438+
// set with each battery at or above its own count. A set difference names
439+
// WHICH battery stopped running; a count says only that something did.
440+
//
441+
// ⛔ A merely non-zero count is not the repair, and neither is a pinned TOTAL:
442+
// a battery dropping from 9 cases to 3 keeps a total "right" for the wrong
443+
// reason the moment a sibling grows.
444+
//
445+
// The counts are a FLOOR, not an equality: adding cases is ordinary work and
446+
// must not red. A battery BELOW its floor means cases stopped running, and the
447+
// remedy is to find what stopped registering -- never to lower the number.
448+
//
449+
// Measured on 597020aa5 by instrumenting `ok` and printing the per-battery
450+
// tally: 7 / 4 / 4 / 4 / 9 / 15, 43 in total.
451+
//
452+
// Five of the six are pinned at what they measure. `live ledger` is not, and
453+
// deliberately: its case count is `1 + one per SELF_TEST_RUN_OTHERWISE row's
454+
// worth`, and that list is ⛔ SHRINK-ONLY by design -- deleting a row is the
455+
// outcome the list exists to reach. A floor at today's 15 would turn every
456+
// legitimate shrink red and train the next author to edit the floor, which is
457+
// the one habit these floors exist to prevent. So what is pinned there is the
458+
// invariant that does NOT move with the list: the structural case ran AND at
459+
// least one row was actually audited (1 + 3). A loop that stops iterating, or
460+
// a ledger that empties without the structural case noticing, still reds.
461+
const SELF_TEST_BATTERIES = Object.freeze({
462+
'comment mask': 7,
463+
'right boundary': 4,
464+
'alias resolution': 4,
465+
'population verdict': 4,
466+
'ledger hygiene': 9,
467+
'live ledger': 4,
468+
});
469+
470+
// DELETING an entry silences that battery's floor exactly as effectively as
471+
// zeroing it, so the registry's own size is pinned too. Adding a battery raises
472+
// this number; removing one is the same ⛔ deliberate edit as lowering a count.
473+
const SELF_TEST_BATTERY_FLOOR = 6;
474+
475+
// The key an assertion is filed under when no battery is open. It is not a
476+
// declared battery, so it reds by the same set difference rather than silently
477+
// inflating whichever battery happened to run last.
478+
const UNATTRIBUTED_BATTERY = '(no battery open)';
479+
480+
// Returned by `selfTest()` only after the floor has been evaluated and the
481+
// verdict printed. The dispatch refuses anything else: a `return` that leaves
482+
// the function early prints nothing and exits 0, which is the same
483+
// nothing-ran-nothing-complained pass one level up.
484+
const SELF_TEST_VERDICT = 'check-self-test-wired self-test reached its verdict';
485+
413486
function selfTest() {
414487
const failures = [];
488+
const seen = new Map();
489+
let openBattery = null;
490+
// Declare the battery the following assertions belong to. The name must be a
491+
// key of SELF_TEST_BATTERIES -- an unknown one reds by set difference,
492+
// naming itself, rather than being counted somewhere it is not floored.
493+
const battery = (name) => {
494+
openBattery = name;
495+
};
415496
const ok = (cond, label) => {
497+
const b = openBattery ?? UNATTRIBUTED_BATTERY;
498+
seen.set(b, (seen.get(b) ?? 0) + 1);
416499
if (!cond) failures.push(label);
417500
};
418501
const wf = (text, name = 'lint.yml') => [{ name, text }];
419502

420503
// ── Prose never decides anything, in either direction ────────────────────
504+
battery('comment mask');
421505
ok(
422506
!carriesSelfTest('scripts/x.mjs', '// run it with --self-test sometimes\nconst a = 1;\n'),
423507
'a `--self-test` that exists only in a JS comment was read as an implementation',
@@ -447,6 +531,7 @@ function selfTest() {
447531
}
448532

449533
// ── Right boundary: the defect class this gate is itself in the family for ─
534+
battery('right boundary');
450535
{
451536
const got = collectInvocations(wf(' - run: node scripts/g.mjs --self-test-extra\n'), {});
452537
ok(got.named.has('scripts/g.mjs'), 'the script was not seen at all — the boundary case would test nothing');
@@ -468,6 +553,7 @@ function selfTest() {
468553
}
469554

470555
// ── Aliases: reached only when a workflow actually names them ────────────
556+
battery('alias resolution');
471557
{
472558
const pkg = {
473559
'check:thing': 'node scripts/thing.mjs --self-test && node scripts/thing.mjs',
@@ -491,6 +577,7 @@ function selfTest() {
491577
}
492578

493579
// ── The population verdict, both directions ──────────────────────────────
580+
battery('population verdict');
494581
{
495582
const carriers = new Set(['scripts/g.mjs']);
496583
const run = (text, ledger = []) => {
@@ -511,6 +598,7 @@ function selfTest() {
511598
}
512599

513600
// ── Ledger hygiene: every row must still be true, and still be needed ────
601+
battery('ledger hygiene');
514602
{
515603
const carriers = new Set(['scripts/w.mjs', 'scripts/t.mjs']);
516604
const sources = {
@@ -561,6 +649,7 @@ function selfTest() {
561649
}
562650

563651
// ── The live ledger, checked against the real tree ───────────────────────
652+
battery('live ledger');
564653
{
565654
const sourceOf = (relPath) => {
566655
try {
@@ -582,18 +671,79 @@ function selfTest() {
582671
}
583672
}
584673

674+
// ── The floor: every declared battery RAN, and ran its cases ─────────────
675+
//
676+
// Evaluated here, after every battery has had its chance and BEFORE the
677+
// verdict -- so the line below can only be printed by a run in which the set
678+
// of batteries that registered assertions equals the set declared.
679+
const declaredBatteries = Object.keys(SELF_TEST_BATTERIES);
680+
const totalCases = [...seen.values()].reduce((a, b) => a + b, 0);
681+
let floorBreached = false;
682+
if (declaredBatteries.length < SELF_TEST_BATTERY_FLOOR) {
683+
floorBreached = true;
684+
failures.push(
685+
`SELF_TEST_BATTERIES declares ${declaredBatteries.length} batteries, below the pinned ` +
686+
`${SELF_TEST_BATTERY_FLOOR} — a battery deleted from the registry takes its own floor with it.`,
687+
);
688+
}
689+
for (const [name, count] of seen) {
690+
if (declaredBatteries.includes(name)) continue;
691+
floorBreached = true;
692+
failures.push(
693+
`self-test battery "${name}" registered ${count} case(s) but is not declared in ` +
694+
'SELF_TEST_BATTERIES — an assertion attributed to no declared battery is one nothing floors.',
695+
);
696+
}
697+
for (const name of declaredBatteries) {
698+
const count = seen.get(name) ?? 0;
699+
if (count >= SELF_TEST_BATTERIES[name]) continue;
700+
floorBreached = true;
701+
failures.push(
702+
count === 0
703+
? `self-test battery "${name}" DID NOT RUN — 0 cases registered, ${SELF_TEST_BATTERIES[name]} pinned. ` +
704+
'The verdict below would have claimed they hold.'
705+
: `self-test battery "${name}" registered ${count} case(s), below its pinned floor of ` +
706+
`${SELF_TEST_BATTERIES[name]} — cases that used to run no longer do.`,
707+
);
708+
}
709+
585710
if (failures.length > 0) {
586711
console.error('check-self-test-wired --self-test FAILED:');
587712
for (const f of failures) console.error(` - ${f}`);
713+
if (floorBreached) {
714+
console.error(
715+
'\nA battery at or below its floor means cases STOPPED RUNNING — the battery is the bug,\n' +
716+
'not the number. Find what stopped registering (an early return, a deleted block, a guard\n' +
717+
'that now skips) and restore it. Raising a floor after ADDING cases is ordinary work;\n' +
718+
`LOWERING one is ${RATCHET_AUTHORITY_MARKER}, not a co-equal option — "the count\n` +
719+
'legitimately moved" and "something stopped running" need different edits, and only a\n' +
720+
'measurement tells them apart.\n',
721+
);
722+
}
588723
process.exit(1);
589724
}
590725
console.log(
591726
`check-self-test-wired --self-test: ${SELF_TEST_RUN_OTHERWISE.length} live ledger row(s) verified, plus the ` +
592-
'comment mask, the right boundary, alias resolution and both audit directions.',
727+
'comment mask, the right boundary, alias resolution and both audit directions' +
728+
` — ${declaredBatteries.length} declared batteries, ${totalCases} cases registered, every battery at or` +
729+
' above its pinned floor.',
593730
);
731+
return SELF_TEST_VERDICT;
594732
}
595733

596734
if (isEntrypoint(import.meta.url)) {
597-
if (process.argv.includes('--self-test')) selfTest();
598-
else main();
735+
if (process.argv.includes('--self-test')) {
736+
// ⛔ Never `selfTest();` bare, and never `return selfTest()`. A `return`
737+
// anywhere above that verdict prints nothing, evaluates no floor and exits
738+
// 0 — the same nothing-ran-nothing-complained pass the battery floor
739+
// refuses, one level up (#13489).
740+
if (selfTest() !== SELF_TEST_VERDICT) {
741+
console.error(
742+
'\n✗ check-self-test-wired self-test: selfTest() returned without reaching its verdict, so no\n' +
743+
'battery floor was evaluated and no success line was printed. Exiting 0 here would report a\n' +
744+
'self-test that never finished as a self-test that passed.\n',
745+
);
746+
process.exit(1);
747+
}
748+
} else main();
599749
}

0 commit comments

Comments
 (0)