Hit on 2026-08-31 while approving the dev-approval gate for #272. It cost two wrong diagnoses and about twenty minutes.
What porch prints
A failing check is reported like this, verbatim, the whole output:
RUNNING CHECKS...
⚠ Check "tests" overridden: timeout: 1200s
✓ build (26.8s)
✗ tests (385.6s)
DEPRECATED `test.poolOptions` was removed in Vitest 4. ...
[codev tests] Another Vitest run owns shared Tower state; waiting.
pr-create: gh pr create returned no PR URL
pr-create: CODEV_PR_TITLE is required
pr-create: CODEV_PR_BODY is required (set it to "" for an empty body)
...
CHECKS FAILED. Cannot approve gate.
Fix the failures and try again.
Five lines, then .... The name of the failing assertion is not among them, and neither is the vitest summary line.
Why those five lines are the worst possible five
They are the first lines of a 385-second run, so they are startup noise: a deprecation warning, a lock-wait notice, and stderr from a test that passes (bugfix-1455-pr-create-concept.test.ts deliberately invokes pr-create.sh with the vars absent and asserts it throws, so those three pr-create: lines are the expected output of a green test).
Read in isolation they look exactly like a check that failed because CODEV_PR_TITLE was unset. I acted on that reading twice, once by exporting the variables and re-running an eight-minute suite for nothing. The builder had to read the protocol and the test source to establish that dev-approval runs only build and tests, and that no pr-create check exists anywhere in PIR.
"Fix the failures and try again" is an instruction that cannot be followed, because the failure is not named.
What would close this
- Print the tail of a failed check, not the head. The failure and the summary are at the end.
- Better, print the lines that identify the failure: the vitest
FAIL lines and the counts. A check runner that knows it runs vitest can say which test failed.
- Say where the full output is. Write it to a file and print the path, so
... has somewhere to point.
- Never truncate the summary line.
Tests 1 failed | 7428 passed is one line and it is the one that matters.
Cost when it bites
Every wrong reading of this output is an eight-minute suite re-run to test the wrong hypothesis, and the suite holds a workspace-wide lock while it runs, so other builders wait behind it.
Hit on 2026-08-31 while approving the
dev-approvalgate for #272. It cost two wrong diagnoses and about twenty minutes.What porch prints
A failing check is reported like this, verbatim, the whole output:
Five lines, then
.... The name of the failing assertion is not among them, and neither is the vitest summary line.Why those five lines are the worst possible five
They are the first lines of a 385-second run, so they are startup noise: a deprecation warning, a lock-wait notice, and stderr from a test that passes (
bugfix-1455-pr-create-concept.test.tsdeliberately invokespr-create.shwith the vars absent and asserts it throws, so those threepr-create:lines are the expected output of a green test).Read in isolation they look exactly like a check that failed because
CODEV_PR_TITLEwas unset. I acted on that reading twice, once by exporting the variables and re-running an eight-minute suite for nothing. The builder had to read the protocol and the test source to establish thatdev-approvalruns onlybuildandtests, and that nopr-createcheck exists anywhere in PIR."Fix the failures and try again" is an instruction that cannot be followed, because the failure is not named.
What would close this
FAILlines and the counts. A check runner that knows it runs vitest can say which test failed....has somewhere to point.Tests 1 failed | 7428 passedis one line and it is the one that matters.Cost when it bites
Every wrong reading of this output is an eight-minute suite re-run to test the wrong hypothesis, and the suite holds a workspace-wide lock while it runs, so other builders wait behind it.