Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,88 @@ jobs:
- run: bun run lint
- run: bun run typecheck
- run: bun run test

# The engineering quality gates on the lines a pull request changes (issue #20, .quality/gate.py,
# .quality/README.md). Only the first two steps (checkout and the render of the description) get the token,
# and both run before any code from the pull request (bun install is the first); no later step or post step
# gets it.
quality:
if: github.event_name == 'pull_request'
permissions:
contents: read
pull-requests: read
runs-on: ubuntu-latest
# Mutation stops itself inside a 30-minute budget and still writes its report; setup stays well under the rest.
timeout-minutes: 45
steps:
- name: Check out the pull request
# Not actions/checkout: it keeps github.token for its post step, which runs at the end of the job, after
# code from the pull request. This fetches GitHub's merge commit with full history (the gate diffs against
# its first parent and checks the base out), passes the token to git through the environment only for the
# fetch, and stores no credential.
env:
GH_TOKEN: ${{ github.token }}
REPO_URL: ${{ github.server_url }}/${{ github.repository }}
MERGE_REF: ${{ github.ref }}
run: |
set -o pipefail
find . -mindepth 1 -maxdepth 1 -exec rm -rf {} +
git init --quiet .
auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 | tr -d '\n')
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraheader GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" \
git fetch --quiet --no-tags "$REPO_URL" "+$MERGE_REF:refs/remotes/pull/merge"
git checkout --quiet --detach refs/remotes/pull/merge
- name: Render the pull request description
# The gate reads "Equivalent mutant:" waivers from GitHub's own rendering of the description (POST
# /markdown, gfm), so a waiver counts only where a reviewer sees it on the PR page. Fetched now rather than
# taken from the event, so re-running this job after editing the description sees the edit. python3 -I
# keeps the checkout off sys.path. The HTML's sha256 is a step output, which no later step can change.
id: description
env:
GH_TOKEN: ${{ github.token }}
PR_URL: ${{ github.event.pull_request.url }}
API_URL: ${{ github.api_url }}
run: |
set -o pipefail
curl --proto '=https' --tlsv1.2 --retry 5 --fail --silent --show-error \
--header "Authorization: Bearer $GH_TOKEN" --header "Accept: application/vnd.github+json" "$PR_URL" \
| python3 -I -c 'import json, sys; json.dump({"text": json.load(sys.stdin)["body"] or "", "mode": "gfm"}, sys.stdout)' \
| curl --proto '=https' --tlsv1.2 --retry 5 --fail --silent --show-error \
--header "Authorization: Bearer $GH_TOKEN" --header "Accept: application/vnd.github+json" \
--header "Content-Type: application/json" --data-binary @- "$API_URL/markdown" \
> "$RUNNER_TEMP/pr-body.html"
echo "sha256=$(sha256sum "$RUNNER_TEMP/pr-body.html" | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"
- name: No sudo for the gate
# The gate runs code from the pull request and every mutant of it, and none of that may run as root. The
# hosted runner's passwordless sudo rule is dropped, and the job fails if sudo still works.
run: |
if command -v sudo >/dev/null 2>&1; then
sudo -n rm -f /etc/sudoers.d/runner 2>/dev/null || true
if sudo -n true 2>/dev/null; then echo "sudo still works"; exit 1; fi
fi
# Both actions default `token` to github.token; with an empty token they send no auth.
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.4.2
no-cache: true
token: ""
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
token: ""
- run: bun install --frozen-lockfile
- name: Install quality tools
# Pinned commits and versions in .quality/requirements.txt, in a venv that is thrown away with the job.
run: |
python -m venv "$RUNNER_TEMP/quality-venv"
"$RUNNER_TEMP/quality-venv/bin/pip" install --quiet -r .quality/requirements.txt
- name: Gate self-tests
run: |
export PATH="$RUNNER_TEMP/quality-venv/bin:$PATH"
python -m unittest discover -s .quality -p 'test_*.py'
- name: Quality gate
env:
DESCRIPTION_SHA256: ${{ steps.description.outputs.sha256 }}
run: |
export PATH="$RUNNER_TEMP/quality-venv/bin:$PATH"
python .quality/gate.py --description-html "$RUNNER_TEMP/pr-body.html" --description-sha256 "$DESCRIPTION_SHA256"
89 changes: 89 additions & 0 deletions .github/workflows/mutation-baseline.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: mutation-baseline

# The full mutation score of src/ for .quality/baseline.json. Never part of an ordinary pull request:
# a PR is judged on its changed lines by the quality job in ci.yml. It runs by hand, or on a pull request that changes
# the baseline script, and always measures the BASE branch's src/ code (the PR supplies the tests and the gate).
# Result: the RESULT line in the log and the step summary; copy the figures and this run's URL into baseline.json.

on:
workflow_dispatch:
pull_request:
paths:
- ".quality/full_baseline.py"
- ".github/workflows/mutation-baseline.yml"

permissions:
contents: read

# A running baseline is never cancelled: a path filter on pull_request matches the whole PR's diff, so every push to
# such a PR starts this workflow again. The new run waits, and exits at once unless that push changed these files.
concurrency:
group: mutation-baseline-${{ github.ref }}
cancel-in-progress: false

jobs:
baseline:
runs-on: ubuntu-latest
# The script stops itself at 330 minutes and still prints its totals; GitHub's own limit is 360.
timeout-minutes: 350
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
persist-credentials: false
- name: Did this push change the baseline?
id: wanted
env:
ACTION: ${{ github.event.action }}
BEFORE: ${{ github.event.before }}
AFTER: ${{ github.event.pull_request.head.sha }}
run: |
set -eu
run=true
if [ "$ACTION" = synchronize ] && changed=$(git diff --name-only "$BEFORE" "$AFTER" 2>/dev/null); then
printf '%s\n' "$changed" | grep -qxE '\.quality/full_baseline\.py|\.github/workflows/mutation-baseline\.yml' || run=false
fi
echo "run=$run" | tee -a "$GITHUB_OUTPUT"
- name: Measure the base branch's code
if: github.event_name == 'pull_request' && steps.wanted.outputs.run == 'true'
run: |
set -eu
# The source under measurement is the base's; the tests and the gate are the pull request's. The base is the
# merge commit's first parent, the tip the merge was built on (the event's base sha can be older than that).
BASE_SHA=$(git rev-parse HEAD^1)
rm -rf src
git checkout "$BASE_SHA" -- src
echo "src differs from $BASE_SHA in:"; git diff --stat "$BASE_SHA" -- src
- name: No sudo for the measured code
# The hosted runner's passwordless sudo rule is dropped, and the job fails if sudo still works.
if: steps.wanted.outputs.run == 'true'
run: |
if command -v sudo >/dev/null 2>&1; then
sudo -n rm -f /etc/sudoers.d/runner 2>/dev/null || true
if sudo -n true 2>/dev/null; then echo "sudo still works"; exit 1; fi
fi
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
if: steps.wanted.outputs.run == 'true'
with:
bun-version: 1.4.2
no-cache: true
token: ""
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
if: steps.wanted.outputs.run == 'true'
with:
python-version: "3.12"
token: ""
- if: steps.wanted.outputs.run == 'true'
run: bun install --frozen-lockfile
- name: Install quality tools
if: steps.wanted.outputs.run == 'true'
run: |
python -m venv "$RUNNER_TEMP/quality-venv"
"$RUNNER_TEMP/quality-venv/bin/pip" install --quiet -r .quality/requirements.txt
- name: Mutate every site
if: steps.wanted.outputs.run == 'true'
env:
BASELINE_BUDGET_SECONDS: "19800"
run: |
export PATH="$RUNNER_TEMP/quality-venv/bin:$PATH"
python .quality/full_baseline.py
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ coverage/
.claude/settings.local.json
*.log
target/
__pycache__/
106 changes: 106 additions & 0 deletions .quality/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Quality gate

The `quality` job in `.github/workflows/ci.yml` runs on every pull request and calls `gate.py`. It follows the
engineering quality standard as a ratchet: only code the PR adds or changes is judged, so debt already on main never
blocks a PR. Everything lives in this folder, that job and `mutation-baseline.yml`, so one revert removes it.

| Check | Fails when | Tool |
|---|---|---|
| Coverage | an executable line the PR adds or changes has no test hit, or a changed file is missing from the report | Vitest, v8 coverage (lcov) |
| CRAP | a function the PR adds or changes scores 10 or more, unless it was already 10 or more on base and scores no higher now | crapper |
| Duplication | the PR adds a structural duplicate pair at 0.8 similarity or above | dryer |
| Mutation | a mutant on a changed line survives, mutator fails on a file, or the budget runs out | mutator |

The thresholds (CRAP 10, similarity 0.8, a mutation budget of 5 to 30 minutes) are constants in `gate.py`, and a
self-test pins them. Changing one is an owner decision recorded in the engineering standard.

The gate is a port of the one in the sibling Bun repository (buildboard), adapted to Vitest: one scope, Vitest's
coverage, and a mutant command that copes with Vite resolving symbolic links (see Mutation).

## Scope

TypeScript under `src/`, minus tests and `.d.ts`. The test helpers under `test/`, the fake app server included, are
not judged.

A pure-deletion hunk judges no line. Every diff runs with `--no-color --no-ext-diff --no-textconv`, so git settings
cannot change what the gate reads, and a file whose hunks do not add up to git's own count stops the gate.

## Coverage

The gate runs `bun run coverage` itself (with the lcov report pinned), after deleting any old report, so a stale
report never stands in for it. `vitest.config.ts` lists every file under `src/`, so a file no test loads is in the
report at 0 hits.

The CLI entry points, `src/cli/bin.ts` and `src/conformance/cli.ts`, only run as child processes in the tests, which
v8 coverage does not see. Each changed line in them is reported on its own and can be waived (see Waivers). Keep them
thin: the logic belongs in `main.ts` and `runner.ts`, which the tests import.

The repo total (lines and functions, `src/` only) is compared with `baseline.json` and reported. A drop alone does
not fail the PR, because deleting covered code can cause it. A PR that raises a number updates `baseline.json`.

The gate and every mutant run with `TETHER_KEYCHAIN=off` and without `TETHER_REAL_SERVICE`, so nothing it runs
touches the OS keychain or installs a real launchd agent.

## CRAP ratchet

A changed function that already scores 10 or more on base passes when its CRAP is no higher than on base; the report
lists it as "still over the limit" with both scores. A new, moved or renamed function, and one that was under 10 on
base, must stay under 10. Functions match by file, name and order among same-named functions in the file; when the
count of a name in a file changes, every function of that name counts as new. A base function with no coverage
record is not scored (0% would be the highest base possible), so it is judged as new.

The base score is measured, not stored, and only when a touched function is over 10: the gate checks out base in a
throwaway worktree that borrows this checkout's `node_modules`, and runs the same coverage there.

## Mutation

Mutation runs only after every other check passed, on the changed lines of the changed files. The standard's
example command, `mutator --changed --test-command "bun run test"`, does not work here for two reasons, so the gate
calls mutator per file with `--lines` and its own test command, `mutant-tests.sh`:

- mutator runs each mutant in a worker under `target/mutation-workers` made of symbolic links to the tree and a copy
of the mutated file. Vite resolves every module to its real path, so a linked test would import the original file
and every mutant would survive. `mutant-tests.sh` first replaces the links with real copies (all but
`node_modules`).
- The whole suite takes about 45 seconds. The script runs `vitest related` for the file first (the tests that import
it, directly or not), on one Vitest worker, and the whole suite only for a mutant that survived: the entry points
are reached only through child processes, which `related` cannot see.

A mutant is killed only when Vitest reports a failed test or a test file that failed to load. A run that writes no
report (a broken config, a crash) is a harness error and never a kill: the mutant shows as a survivor, and the gate
fails where a reviewer can see it.

The budget is sized to the change: the time already spent, plus for each file the rounds its sites need on its
workers (plus mutator's own unmutated run), times the measured time of the whole suite. It is never below 5 minutes
and never above 30. When it runs out, the check fails cleanly and still reports what finished and what was not
checked; the job timeout (45 minutes) sits above the largest budget.

The quality job removes the hosted runner's sudo rule before any code from the pull request runs, so no mutant can
run anything as root.

## Waivers

A surviving mutant or an untested changed line passes only with this line in the PR description, as a paragraph of
its own:

Equivalent mutant: src/file.ts:42 why no test can tell the difference

One line waives every mutant on that line. The gate reads GitHub's own rendering of the description, fetched and
hashed before any code from the pull request runs, so a waiver counts only where a reviewer sees it. After editing
the description, re-run the quality job: it reads the description as it is when the job starts.

## Baseline

`baseline.json` holds the default branch's numbers. Line coverage moves by about 0.06 points between runs of the
same code (some paths depend on timing), which is one more reason a drop alone never fails a PR: coverage, functions measured, functions over CRAP 10, duplicate
pairs, and the full mutation score from `full_baseline.py`, which mutates every site under `src/` with the gate's own
machinery. `.github/workflows/mutation-baseline.yml` runs it on a GitHub runner, by hand or on a pull request that
changes the script or that workflow, always on the base branch's `src/`. Copy its `RESULT` line and the run's URL
into `baseline.json`.

## Running it locally

python3.12 -m venv /tmp/qv && /tmp/qv/bin/pip install -r .quality/requirements.txt
PATH=/tmp/qv/bin:$PATH python3.12 -m unittest discover -s .quality -p 'test_*.py'
PATH=/tmp/qv/bin:$PATH python3.12 .quality/gate.py --base origin/main
PATH=/tmp/qv/bin:$PATH python3.12 .quality/full_baseline.py
17 changes: 17 additions & 0 deletions .quality/baseline.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"measuredOn": "main at 41d3e6c, 2026-11-18",
"coverage": {
"lines": 97.11,
"functions": 98.96
},
"functionsMeasured": 242,
"functionsOverCrap10": 0,
"duplicatePairs": 0,
"mutation": {
"score": 81.3,
"killed": 514,
"survived": 118,
"sites": 636,
"run": "https://github.com/farzai/tether/actions/runs/44246737986"
}
}
Loading
Loading