Run the full macOS suite on main and nightly, with a one-job macOS gate on PRs into main #893

Activity

1. JacobStephens2

   added

   enhancementNew feature or requestNew feature or request

   on Oct 20, 2026
2. JacobStephens2 commented on Oct 20, 2026

   

   JacobStephens2

   on Oct 20, 2026

   Last edited by JacobStephens2

   OwnerAuthor

   More actions

   This was generated by AI during triage.

   Agent Brief

   Updated 2026-10-20 to widen the scope after the research in #902.

   Category: enhancement
   Summary: Run the full macOS suite on pushes to main and nightly, give PRs into main a one-job macOS gate, and run no macOS on PRs into other bases

   Current behavior:
   The CI workflow runs test-macos-shard (four nextest hash partitions on macos-26) for every pull_request, whatever its base, and for pushes to main. The test-macos aggregate runs with always() and passes only when the shards' result is success, so a skipped shard fails it. Every macOS job shares GitHub's 5-job macOS cap for a personal account. A full run takes 4 of those slots for about 18 minutes, and the measured median from job creation to the last shard is 33 minutes.

   Desired behavior:

   Event	macOS work
   Push to main	Full 4-shard suite, as now
   Nightly schedule on main	Full 4-shard suite
   workflow_dispatch	Full 4-shard suite on the chosen ref
   PR into main without the test:macos label	One gate job (below); no shards
   PR into main with the test:macos label	Full 4-shard suite; the gate job may be skipped
   PR into any other base (Spec branch, Base fix base, anything else)	No macOS jobs

   • The gate job is one macos-26 job that runs cargo clippy --all-targets -- -D warnings and a nextest filterset covering the tests of the macOS-specific code: the #[cfg(target_os = "macos")] process-identity and skills-directory code, and the platform branches in child Run handling. Keep the filterset in one named place (the repository's nextest config or the workflow) with a comment saying what it must cover, so new macOS-specific code has an obvious home.
   • Labelling a PR into main with test:macos starts the full suite without a new push, so the workflow's pull_request trigger includes the labeled activity type. Create the test:macos label if it is missing, with a description saying what it does.
   • The test-macos check is still created on every pull request and push, under that name. It passes when it should, given the table:
     • the shards passed;
     • or the shards were skipped by the rule and the gate passed;
     • or, on a PR into a non-main base, both were skipped.
       It fails when a job that ran failed or was cancelled, or when a job the rule requires was skipped.
   • Linux checks (test, site-test, console-test) are unchanged and run on every pull request.
   • A failing nightly run is visible to the Day shift. GitHub notifies the user who last changed the schedule cron; say so in the docs and name what to do when it is red: fix forward on main, or label the fixing PR test:macos.
   • docs/ci-performance.md's workflow policy gives the table, the reasons and the trade-offs, citing this issue and docs/research/macos-ci-speed.md:
     • a macOS-only regression from a Spec's Ticket shows up at the Spec PR;
     • one outside the gate's filterset shows up on main or overnight, not on its PR.

   Key interfaces:

   • The CI workflow's triggers (push, pull_request with labeled, schedule, workflow_dispatch), the shard job's condition, the new gate job and the test-macos aggregate's success rule.
   • The concurrency group and cancel-in-progress policy stay as they are. A labeled event on a PR must not leave an older run's result standing for the newer head.
   • The factory's CI watch accepts success, neutral or skipped checks. Don't change it.
   • The generated Release workflow stays out of bounds: it is regenerated by scripts/generate-release.py, never edited by hand.

   Acceptance criteria:

   • A PR into a non-main branch creates no macOS jobs, and test-macos succeeds.
   • A PR into main without test:macos runs the gate job and no shards. test-macos passes only if the gate passes.
   • Adding test:macos to an open PR into main starts all four shards on its current head, and test-macos then reflects the shards.
   • A push to main, a nightly run and a workflow_dispatch run all four shards, and test-macos fails if any shard fails or is cancelled.
   • The gate's filterset matches at least the tests of the macOS-specific code. The PR body lists the tests it selects (cargo nextest list with the filter) and how long the gate job took.
   • test, site-test and console-test run on every PR as now, and python3 scripts/generate-release.py --check still passes.
   • docs/ci-performance.md records the table, the reasons, the trade-offs and the nightly-failure handling.
   • The PR body shows real runs:
     • this PR's own run (a PR into main: gate only);
     • the same PR after adding test:macos (shards);
     • one PR into a throwaway non-main branch (no macOS jobs, test-macos passing), closed and its branch deleted afterwards.

   Out of scope:

   • Making the tests themselves faster (Find what still makes the macOS test shards slow after #857 #894, and Decide whether Worktree ownership inspection may recheck the launch repository less often #858).
   • The Release workflow's build-local-artifacts (aarch64-apple-darwin) job on pull requests.
   • Third-party or self-hosted macOS runners.
   • Changing how the factory's CI watch reads checks, or how a Spec run delivers its Spec PR.
   • Branch protection or rulesets.
3. JacobStephens2

   added

   ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent

   on Oct 20, 2026
4. JacobStephens2

   changed the title [-]Run the macOS test shards only for pull requests into main[/-] [+]Run the full macOS suite on main and nightly, with a one-job macOS gate on PRs into main[/+] on Oct 20, 2026