From e1ed9366fd40d4854cea51f1e9b87364194fae3d Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Tue, 8 Sep 2026 12:18:23 -0700 Subject: [PATCH] test(cli): stop asserting the winner of a 1ms race in the Vale timeout test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The test gave a 1ms budget to a one-line document and demanded a timeout. That asserts the winner of a race: the timer has to fire before a child running in its OWN process, which does not care whether our event loop is free. Measured, that document takes Vale about 46ms, so 1ms normally wins — and it was seen losing once, across four concurrent full-suite runs, reporting a clean "ok" where the test demanded a "timeout". Captured assertion, which issue #262 had never managed to record for its own two tests: FAIL test/vale-run.test.ts > marks a timeout blocking expected { status: 'ok', blocking: false } to match { status: 'timeout', blocking: true } The production timeout is correct and is not changed: a settled guard, a cleared timer, and an unref so it never holds the loop open. Only the test's budget was wrong. The race is removed by making the work outlast the budget by a margin nothing plausible closes. Vale is QUADRATIC in the size of one file, measured on the pinned binary: 80KB 0.3s | 160KB 0.9s | 320KB 3.5s | 640KB 14s | 1MB 48s So ~320KB of prose against a 100ms budget is a 33x margin the right way round, confirmed by the mutation check: disabling the timer lets the same run finish in 3290ms and report "ok". Eight concurrent copies of the test now pass, which is the contention that broke the old one. The run is killed at 100ms, so the test costs about that rather than 3.5s. This is not either of the two tests #262 names — those passed 9 of 9 runs here. It is a separate flake that entered with 54cd0c0, inside the 0.11.2 range. Refs #262 --- packages/cli/test/vale-run.test.ts | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/packages/cli/test/vale-run.test.ts b/packages/cli/test/vale-run.test.ts index 028f80a4..dea2028c 100644 --- a/packages/cli/test/vale-run.test.ts +++ b/packages/cli/test/vale-run.test.ts @@ -431,14 +431,32 @@ withVale("ValeRunOutcome.blocking against the real binary", () => { // Vale was present and asked to work. Reporting this as a skip would let a // broken rule file read as "no Vale findings" — indistinguishable from a // clean run, and how a silently disabled engine ships. + // + // THE BUDGET AND THE INPUT ARE BOTH LOAD-BEARING, and an earlier version + // of this test got it wrong. It gave a 1ms budget to a one-line document, + // which asserts the winner of a race: the timer has to fire before a child + // that runs in its OWN process and does not care whether our event loop is + // free. Measured, that document takes Vale about 46ms, so 1ms normally + // wins — but under load the timer's callback is delayed while the child + // keeps going, and it was seen losing once across four concurrent + // full-suite runs, reporting a clean "ok" where the test demanded a + // "timeout". + // + // The race is removed by making the work outlast the budget by a margin + // nothing plausible closes. Vale is QUADRATIC in the size of a single + // file — measured on the pinned binary at 80KB 0.3s, 160KB 0.9s, 320KB + // 3.5s, 640KB 14s — so roughly 320KB of prose takes about 3.5 SECONDS + // against a 100ms budget. That is a 35x margin the right way round, where + // the old one was a 46x margin the wrong way. The run is killed at 100ms, + // so the test costs about that rather than 3.5s. const cwd = makeProject( `${header}\n[*.md]\nno-simply.no-simply = YES\n`, { "no-simply": existenceRule("simply", "Avoid 'simply'") }, - { "doc.md": "Just simply do it.\n" } + { "doc.md": `${"Just simply do it. ".repeat(17_000)}\n` } ); expect( - await runVale({ cwd, paths: ["doc.md"], timeoutMs: 1 }) + await runVale({ cwd, paths: ["doc.md"], timeoutMs: 100 }) ).toMatchObject({ status: "timeout", blocking: true }); });