Skip to content

ci: adopt the engineering quality gates - #21

Merged
parsilver merged 7 commits into
mainfrom
ci/20-quality-gates
Nov 18, 2026
Merged

parsilver merged 7 commits into
mainfrom
ci/20-quality-gates

Conversation

@parsilver

@parsilver parsilver commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #20

Adds the engineering quality gates as a ratchet: a quality job in ci.yml judges only the lines a pull request adds or changes under src/, so existing debt never blocks a PR. Everything lives in .quality/, that job and mutation-baseline.yml, so one revert removes it.

The gate is a port of the one already running in buildboard (Bun), adapted to Vitest. Review it as a diff against that gate: the logic for diffs, waivers, CRAP ratchet, budget and worker sizing is unchanged; what changed is the scope (src/ only), the coverage command (bun run coverage with a pinned lcov report), the environment the tests run in, and the mutant test command. I based it on buildboard rather than polity because tether is a single package, and polity's gate is mostly per-package Vitest project routing that tether does not need; the Vite symlink fix comes from polity.

Acceptance criteria

  • Quality job on every PR, on ubuntu-latest, about 5 minutes. quality runs on pull_request only; no path filter. On this PR it took 1m23s (self-tests 17s, suite with coverage 40s). Mutation adds time only when src/ lines change: up to a budget sized to the change, 5 to 30 minutes.
  • CRAP. Fails when a touched function scores 10 or more. A function already at 10 or more on base passes when it is no higher (the base is measured in a throwaway worktree, only when needed); untouched functions are not judged. Self-tests: CrapVerdict, CrapFailure, CheckCrap, CrapBase.
  • Duplication. Fails when a dryer pair at 0.8 or above touches a changed line and its two files hold more pairs than on base. Self-tests: DuplicateVerdict, DryerOutput, DryerRun (the real dryer).
  • Mutation. Changed lines of changed files only, with a budget that fails cleanly and still prints the partial report. A survivor, or a changed executable line with no test hit, fails unless the PR description waives it line by line (Equivalent mutant: src/file.ts:42 reason, read from GitHub's own rendering, fetched and hashed before any PR code runs). Self-tests: MutationBudget, BudgetSizing, Workers, EquivalentWaivers, ReadWaivers, MutantTestsScript (real Vitest).
    The command is not the literal mutator --changed --test-command "bun run test", for two reasons. First, mutator's workers are symbolic links, and Vite resolves links to real paths, so every mutant would survive. mutant-tests.sh copies the links first. Second, the full suite takes about 45 seconds, so the script runs vitest related first and the whole suite only for a survivor. The CLI entry points only run as child processes, which related cannot see.
  • Coverage. Reported on every PR. Changed executable lines must be hit, and the repo total is compared with the baseline (a drop caused by deleting code does not fail). Self-tests: UncoveredChangedLines, LcovTotals, CoverageVerdict, ChangedRanges (pure-deletion hunks judge nothing).
  • .quality/baseline.json measured on main at 41d3e6c: lines 97.11%, functions 98.96%, 242 functions, none over CRAP 10, no duplicate pair, mutation score 81.3% (514 killed, 118 survived, 636 sites, every file checked, run, 76 minutes). The Baseline self-test fails if a number is missing.
  • No threshold lowered. Thresholds pins CRAP 10, similarity 0.8 and the 5 to 30 minute budget.

Changes from the fresh-eyes plan review

  • A file no test imports (vitest related finds nothing) runs --passWithNoTests and then the whole suite, so its mutants are not false kills.
  • A harness error (no JSON report: a broken config, a crash) is never a kill. The mutant shows as a survivor, so the gate fails visibly.
  • The gate runs with TETHER_KEYCHAIN=off and without TETHER_REAL_SERVICE. Every mutant also gets its own TETHER_HOME, so nothing touches the keychain, a real launchd agent or ~/.tether.
  • Token handling, copied from buildboard: only the checkout and the description render get the token, both before bun install. No actions/checkout post step. token: "" on setup-bun and setup-python. Permissions contents: read and pull-requests: read. The hosted runner's sudo rule is dropped before the gate. QualityWorkflow self-tests pin all of it.

Evidence

  • Self-tests on CI: Ran 128 tests in 16.723s, OK. Locally (macOS, load average above 250): 128 tests, OK apart from the workflow and baseline tests, which failed before those files existed.
  • The gate on this PR, on CI: Waivers pass; Coverage pass ("Every executable line the PR adds or changes is hit by a test"); CRAP, Duplication and Mutation pass ("No lines were added or changed in TypeScript under src/").
  • End to end on a throwaway commit that changes src/protocol/json.ts:3 (text === "" to text.length === 0), run locally with gate.py --base HEAD~1: coverage, CRAP (1 function scored) and duplication passed. Mutation killed 1 mutant and reported 1 survivor, 0 -> 1. That survivor is equivalent: JSON.parse("") throws, and the catch returns undefined anyway. It is the kind of case a waiver exists for. The mutant run took 243s of a 300s budget.
  • bun run lint: clean. No TypeScript changed, so typecheck and the Vitest suite are unaffected; CI's check jobs run them.
  • The gate's own Python: dryer finds no duplicate pair at 0.8. The highest cyclomatic complexity is 9 (crap_verdict).

Risks

  • Flaky tests under load. A flaky failure during a mutant run reads as a kill, which makes the gate too lenient, never too strict. A flaky failure in mutator's unmutated run fails that file, and re-running fixes it. Two flakes seen while building this, neither touched by this PR:
    • test/cli.test.ts "kills the agent on timeout", locally under heavy load;
    • test/service-real.test.ts:93 on the macOS check job, where launchctl print is checked right after the bootout.
  • Mutation time on wide files. A PR that changes many sites in src/cli/main.ts or src/client/worker.ts can need most of the 30-minute budget: in the full run those files took about 10 to 13 minutes on 4 workers. Above the budget, the standard says to split the PR or speed up its tests.
  • Line coverage moves by about 0.06 points between runs of the same code (CI measured 97.17% once and 97.11% once; some paths depend on timing). A drop alone does not fail, so this only affects the report.
  • The 118 surviving mutants on main are existing debt and do not block any PR. They are worth a follow-up issue.

Not verified

  • A pull request that changes src/ lines has not gone through the gate on CI yet. That path was only run locally (above).
  • The waiver path on CI with a real description: covered by self-tests on recorded GitHub renderings, not by a live PR.

…t command

Port the changed-line quality gate from the sibling Bun repository and
adapt it to Vitest: one scope (src/), the package coverage script with a
pinned lcov report, and a mutant command that copies mutator's linked
worker tree so Vite loads the mutated file. The workflow job and the
baseline do not exist yet, so their self-tests fail (9 errors, 119 pass).
@parsilver parsilver mentioned this pull request Nov 18, 2026
7 tasks
One quality job on ubuntu-latest, pull requests only. The token reaches
only the checkout and the render of the description, both before any
code from the pull request runs; sudo is dropped before the gate. The
README says what each check judges and why mutation uses its own test
command. Workflow self-tests pass; only the baseline test still fails.
Coverage, functions over CRAP 10 and duplicate pairs measured on main at
41d3e6c (242 functions, none over 10, no pair at 0.8). The full mutation
run (636 sites) runs on a hosted runner through mutation-baseline.yml;
its score fills in baseline.json once that run finishes.
Biome formats YAML too, and lint failed on the single-quoted strings in
the new jobs. The workflow self-test now looks for the double-quoted
empty token.
514 of 632 covered mutants killed (81.3%) over all 636 sites of src/,
every file checked, measured by mutation-baseline.yml on the hosted
runner against main at 41d3e6c.
The gate compares coverage on ubuntu-latest, where main's src measures
97.17% of lines (macOS locally: 97.11%, platform-specific paths). The
gate's first CI run asked for this update.
A second CI run of the same code measured 97.11%, not 97.17%: line
coverage moves by about 0.06 points between runs because some paths
depend on timing, not on the platform as the previous commit assumed.
@parsilver
parsilver marked this pull request as ready for review November 18, 2026 07:11

@parsilver parsilver left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Blocker. A port of buildboard's gate to Vitest, scoped to src/ as a ratchet on changed lines.

The quality job runs on pull_request, not pull_request_target, with contents: read and pull-requests: read. The token is used only in the first two steps, the manual fetch of the merge ref and the render of the description, before any PR code runs. Nothing stores a credential, and actions/checkout is skipped so its post step can't run with the token after PR code. Sudo is dropped and checked before the gate runs PR code and its mutants. Waivers count only from GitHub's own rendering of the description, so a waiver counts only where a reviewer can see it. The full baseline runs only by hand or on changes to its own script, measures the base branch, and is never cancelled. Hosted runners are fine here because the repo is public. All checks are green, including the gate on itself. Merging.

@parsilver
parsilver merged commit 3f52e43 into main Nov 18, 2026
4 checks passed
@parsilver
parsilver deleted the ci/20-quality-gates branch November 18, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt engineering quality gates

1 participant