Repository navigation
ci: adopt the engineering quality gates - #21
Conversation
…t command Port the changed-line quality gate from the sibling Bun repository and adapt it to Vitest: one scope (src/), the package coverage script with a pinned lcov report, and a mutant command that copies mutator's linked worker tree so Vite loads the mutated file. The workflow job and the baseline do not exist yet, so their self-tests fail (9 errors, 119 pass).
One quality job on ubuntu-latest, pull requests only. The token reaches only the checkout and the render of the description, both before any code from the pull request runs; sudo is dropped before the gate. The README says what each check judges and why mutation uses its own test command. Workflow self-tests pass; only the baseline test still fails.
Coverage, functions over CRAP 10 and duplicate pairs measured on main at 41d3e6c (242 functions, none over 10, no pair at 0.8). The full mutation run (636 sites) runs on a hosted runner through mutation-baseline.yml; its score fills in baseline.json once that run finishes.
Biome formats YAML too, and lint failed on the single-quoted strings in the new jobs. The workflow self-test now looks for the double-quoted empty token.
514 of 632 covered mutants killed (81.3%) over all 636 sites of src/, every file checked, measured by mutation-baseline.yml on the hosted runner against main at 41d3e6c.
The gate compares coverage on ubuntu-latest, where main's src measures 97.17% of lines (macOS locally: 97.11%, platform-specific paths). The gate's first CI run asked for this update.
A second CI run of the same code measured 97.11%, not 97.17%: line coverage moves by about 0.06 points between runs because some paths depend on timing, not on the platform as the previous commit assumed.
parsilver
left a comment
There was a problem hiding this comment.
No Blocker. A port of buildboard's gate to Vitest, scoped to src/ as a ratchet on changed lines.
The quality job runs on pull_request, not pull_request_target, with contents: read and pull-requests: read. The token is used only in the first two steps, the manual fetch of the merge ref and the render of the description, before any PR code runs. Nothing stores a credential, and actions/checkout is skipped so its post step can't run with the token after PR code. Sudo is dropped and checked before the gate runs PR code and its mutants. Waivers count only from GitHub's own rendering of the description, so a waiver counts only where a reviewer can see it. The full baseline runs only by hand or on changes to its own script, measures the base branch, and is never cancelled. Hosted runners are fine here because the repo is public. All checks are green, including the gate on itself. Merging.
Closes #20
Adds the engineering quality gates as a ratchet: a
qualityjob inci.ymljudges only the lines a pull request adds or changes undersrc/, so existing debt never blocks a PR. Everything lives in.quality/, that job andmutation-baseline.yml, so one revert removes it.The gate is a port of the one already running in buildboard (Bun), adapted to Vitest. Review it as a diff against that gate: the logic for diffs, waivers, CRAP ratchet, budget and worker sizing is unchanged; what changed is the scope (
src/only), the coverage command (bun run coveragewith a pinned lcov report), the environment the tests run in, and the mutant test command. I based it on buildboard rather than polity because tether is a single package, and polity's gate is mostly per-package Vitest project routing that tether does not need; the Vite symlink fix comes from polity.Acceptance criteria
ubuntu-latest, about 5 minutes.qualityruns onpull_requestonly; no path filter. On this PR it took 1m23s (self-tests 17s, suite with coverage 40s). Mutation adds time only whensrc/lines change: up to a budget sized to the change, 5 to 30 minutes.CrapVerdict,CrapFailure,CheckCrap,CrapBase.DuplicateVerdict,DryerOutput,DryerRun(the real dryer).Equivalent mutant: src/file.ts:42 reason, read from GitHub's own rendering, fetched and hashed before any PR code runs). Self-tests:MutationBudget,BudgetSizing,Workers,EquivalentWaivers,ReadWaivers,MutantTestsScript(real Vitest).The command is not the literal
mutator --changed --test-command "bun run test", for two reasons. First, mutator's workers are symbolic links, and Vite resolves links to real paths, so every mutant would survive.mutant-tests.shcopies the links first. Second, the full suite takes about 45 seconds, so the script runsvitest relatedfirst and the whole suite only for a survivor. The CLI entry points only run as child processes, whichrelatedcannot see.UncoveredChangedLines,LcovTotals,CoverageVerdict,ChangedRanges(pure-deletion hunks judge nothing)..quality/baseline.jsonmeasured on main at 41d3e6c: lines 97.11%, functions 98.96%, 242 functions, none over CRAP 10, no duplicate pair, mutation score 81.3% (514 killed, 118 survived, 636 sites, every file checked, run, 76 minutes). TheBaselineself-test fails if a number is missing.Thresholdspins CRAP 10, similarity 0.8 and the 5 to 30 minute budget.Changes from the fresh-eyes plan review
vitest relatedfinds nothing) runs--passWithNoTestsand then the whole suite, so its mutants are not false kills.TETHER_KEYCHAIN=offand withoutTETHER_REAL_SERVICE. Every mutant also gets its ownTETHER_HOME, so nothing touches the keychain, a real launchd agent or~/.tether.bun install. Noactions/checkoutpost step.token: ""on setup-bun and setup-python. Permissionscontents: readandpull-requests: read. The hosted runner's sudo rule is dropped before the gate.QualityWorkflowself-tests pin all of it.Evidence
Ran 128 tests in 16.723s, OK. Locally (macOS, load average above 250): 128 tests, OK apart from the workflow and baseline tests, which failed before those files existed.src/protocol/json.ts:3(text === ""totext.length === 0), run locally withgate.py --base HEAD~1: coverage, CRAP (1 function scored) and duplication passed. Mutation killed 1 mutant and reported 1 survivor,0 -> 1. That survivor is equivalent:JSON.parse("")throws, and the catch returns undefined anyway. It is the kind of case a waiver exists for. The mutant run took 243s of a 300s budget.bun run lint: clean. No TypeScript changed, so typecheck and the Vitest suite are unaffected; CI'scheckjobs run them.crap_verdict).Risks
test/cli.test.ts"kills the agent on timeout", locally under heavy load;test/service-real.test.ts:93on the macOScheckjob, wherelaunchctl printis checked right after the bootout.src/cli/main.tsorsrc/client/worker.tscan need most of the 30-minute budget: in the full run those files took about 10 to 13 minutes on 4 workers. Above the budget, the standard says to split the PR or speed up its tests.Not verified
src/lines has not gone through the gate on CI yet. That path was only run locally (above).