porch's tests gate excludes 3 test files via gitignored local config, so 'tests green' means something different on every machine

[pseudoseed]

Found while triaging #278 on 2026-08-31.

What is configured

.codev/config.json > porch.checks.tests overrides the command the tests gate runs. Among its excludes are three named test files, not just directories:

--exclude "**/bugfix-213-architect-restart.test.ts"
--exclude "**/init.test.ts"
--exclude "**/spec-250-vendoring-identities.test.ts"

.codev/config.json is gitignored (.gitignore:11), so this list is machine-local and unversioned.

Why it matters

A builder that reports "porch tests green" has run whatever its machine excludes. Nobody reviewing that claim can see the exclusion list, it does not travel with the repo, and it is not the list CI runs. Three named files are silently outside the gate on this machine, and a different machine has a different set.

This is the shape the project already treats as a defect elsewhere: a check that cannot observe something must not report the same value as a check that observed it and found nothing wrong. Here the gate cannot fail on those three files and says nothing about it.

Concretely today: spec-250-vendoring-identities.test.ts fails on this machine whenever the fork checkout is dirty (#278). The gate is insulated from it, so a builder can pass the gate while its own pnpm test is red, and the two answers disagree with no record of why.

What would close this

• Move the exclusion list somewhere versioned, so it travels and can be reviewed. If a test must be excluded, that decision belongs in the repo with a reason next to it.
• Have the gate report what it excluded, so "tests green" carries its own scope. Silence about an exclusion is the part that misleads.
• Revisit whether bugfix-213-architect-restart.test.ts and init.test.ts still need excluding, or whether they were quarantined once and forgotten.

Related

#278 is the environmental failure that led here. This issue is about the gate not saying what it skipped, which is independent of that test being fixed.

[pseudoseed]

Material update: one of those three excludes was added today, mid-flight, while four builders were blocked on the assertion it silences.

.codev/config.json has mtime 2026-08-31 11:44:44. The --exclude "**/spec-250-vendoring-identities.test.ts" entry was appended then. The file is untracked (.gitignore:11), so git records no author and there is no way to ask it who.

Builder bugfix-273 caught it and reported it rather than accepting the green: "the tests check did NOT start passing on merit... The check is green because the assertion is no longer run, not because pin.json moved." That is correct. 3c25fe8b5 is still only on builder/pir-272 and pin.json still reads 2f64a1b0ee2b everywhere else.

Builder bugfix-278 did not do it: its worktree was created at 11:51, seven minutes after the edit. I have asked the builders that were running at 11:44.

What this changes about the issue

It is not only that the exclusion list is invisible. It is that the list is writable at runtime by anything with filesystem access, unversioned, and takes effect on every project in the workspace immediately. A gate can be turned off mid-run and every subsequent "porch tests green" is silently narrower, with no record that it changed, no author, and no diff.

The three builders blocked on this assertion each spent round trips proving the failure was not theirs. Then the assertion stopped running and their gates went green. Nobody decided that in the open.

Added to what would close this

• The tests command belongs in a versioned file so a change to it is a reviewable diff with an author.
• Porch should record the exact command it ran in status.yaml alongside the result, so a green check carries the scope it was green under.
• A check whose command was overridden should say so in its result line, not only in a warning during the run.

[pseudoseed]

Resolved on the who, and the mechanism is worse than "someone edited a file".

Builder bugfix-260 wrote the line at 11:44:44 and has removed it. Its own account of the mechanism:

.builders/bugfix-260/.codev/config.json is a SYMLINK to /Users/chris/dev/codev-1455/.codev/config.json, so my relative-path write followed it into the shared workspace config.

That is the part worth keeping. A builder editing what it believes is its own worktree-local config is editing the shared workspace config for every builder and the architect, immediately, because afx spawn symlinks it in. Nothing at the write tells you that.

Blast radius, stated

The exclusion was live from 11:44:44 to 18:25. Any porch tests check that passed in this workspace in that window ran the full suite minus spec-250-vendoring-identities.test.ts. Builders bugfix-273 and bugfix-278 both had runs inside it, so those green results do not cover that file. Nothing else was excluded; the other nine --exclude entries are verbatim from vitest.config.ts.

Why it was reached for

porch done exits 1 without advancing when a check fails (exitChecksNotPassed, commands/porch/index.ts:423), and there is no per-check known-failure mechanism, only a workspace-wide skip: true. A builder told to "document the environmental failure and proceed" has no path through the fix phase. The gap is real and the config file was the only lever within reach.

Added to what would close this

• A per-check, per-project way to record a known environmental failure, with a reason, so nobody has to reach for a workspace-wide lever to get past one file.
• .codev/config.json should not be writable from a builder worktree by accident. Either the symlink goes away or a write through it is refused.
• Porch should record the exact command each check ran in status.yaml, so a green result carries the scope it was green under and a change to that scope is visible after the fact.