Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,11 @@ Decided 2026-10-30 (user, /quest-plan), replacing "First public release":
conflicts (71% of moq's 246 removals); the async fixer now absorbs those.
- Keep the host queue, serial

The [queue watcher](/quest/m0/queue-watcher/README.md) ranks first and
records the full interview: moq first, self-hosted `merge_group` runners,
host-gate work frozen, build concurrency 1, re-enqueue at the back,
unlimited attempts while progressing, flaky re-enqueues, and fixing picks
GitHub refuses. The land work above now means the watcher first, then the
fixer and its host.
The queue watcher epic recorded the full interview: moq first, self-hosted
`merge_group` runners, host-gate work frozen, build concurrency 1,
re-enqueue at the back, unlimited attempts while progressing, and flaky
re-enqueues. Its git history keeps it; only
[its first flaky-test disable](/quest/m0/queue-watcher/README.md) remains.

Decided while planning m0 (2026-09-26), replacing the earlier copied-files and
ownership-manifest design:
Expand Down
192 changes: 40 additions & 152 deletions quest/m0/queue-watcher/README.md
Original file line number Diff line number Diff line change
@@ -1,158 +1,46 @@
# [M] Land through a serial merge queue with an async fixer
# [S] Land the queue watcher's first flaky-test disable on moq-dev/moq

## Goal

Each repository lands through a serial merge queue, one PR at a time, with no
batching and no pre-checks: GitHub's merge queue where the repository has one,
and otherwise the watcher's own queue, such as on private moq-dev/moq.pro,
where GitHub Team offers none. When the queue drops a PR for a failure or a
conflict, a watcher on the maintainer's host runs the existing Codex land
fixer in the background and re-enqueues the PR at the back of the queue. Only
a decision the fixer can't make reaches a person. moq-dev/moq moves first;
moq-dev/quest, rust2 and rind follow once moq proves it, then moq.pro through
the own queue, and the batching host gate (`quest land serve`) is then
retired.

The epic finishes with an end-to-end run on moq-dev/moq: a PR the queue drops
for a failing build is fixed and re-enqueued, a PR dropped for a conflict is
resolved and re-enqueued, a flaky failure is re-enqueued unchanged, a test
seen flaking twice is disabled by a PR that lands through moq's queue with its
`quest/m0/flaky/` epic passing `quest check`, and a design conflict is handed
back. The README's landing section describes the watcher.

Non-goals: moq's own switch (its ruleset, its `merge_group` workflow, the
runner label, and retiring its `quest land` requirement). moq-dev/moq plans
that in its tree, in
[Land main through GitHub's merge queue](https://github.com/moq-dev/moq/blob/main/quest/m0/ci/land.md),
rewritten by [moq-dev/moq#7589](https://github.com/moq-dev/moq/pull/7589).
The queue watcher's end-to-end run on moq-dev/moq finishes with its last
step: a test the land fixer judged flaky in two `merge_group` builds is
disabled by a `quest-land/flaky-<sha>` PR that lands through moq's merge
queue, and moq's `main` passes `quest check` with the `quest/m0/flaky/` epic
and follow-up quest that PR added.

## Plan

Decided 2026-11-01 (maintainer, /quest-plan), reversing the 2026-10-26
decision recorded in moq-dev/moq's `quest/m0/ci/land.md` to leave GitHub's
merge queue for a batched gate. That decision's main reason was that GitHub
cannot resolve quest-file conflicts: 71% of moq's 246 merge-conflict removals
touched `quest/`. The async fixer now absorbs those, resolving them with the
quest merge driver and re-enqueuing. The batched host gate measured about 4-6
merged PRs per hour at a 23% per-PR failure rate, with batches shrinking to
one PR; GitHub's serial queue measured 5.5.

Decision prompts, selected options marked ✅:

- Goal:
- ✅ Yes, as stated: a serialized GitHub merge queue, with every failure
fixed asynchronously and re-enqueued. Batching and pre-checks are
dropped: on moq a useful pre-check must compile, which is most of a check.
- Keep the host queue, serial
- Scope:
- ✅ moq first, since it lands the most PRs and so proves the design
fastest. moq-dev/quest, moq.pro, rust2 and rind stay on the host
`quest land` gate until moq proves it, then
follow (done 2026-11-03; proven from a fork on the PR that deleted it).
- All five at once
- moq and quest
- Runners:
- ✅ Self-hosted on the maintainer's host: `merge_group` builds run on its
runner (label `moq-queue`), for warm caches and no GitHub minutes
- GitHub-hosted
- Interim:
- Keep improving the host gate
- ✅ Freeze gate work: no more host-gate improvements except fixes the
switch needs. Closed for it: [moq-dev/quest#376](https://github.com/moq-dev/quest/pull/376)
(batch sizing), [#379](https://github.com/moq-dev/quest/pull/379) (batch
quest check), [#375](https://github.com/moq-dev/quest/pull/375) (rerun
before blame), [#377](https://github.com/moq-dev/quest/pull/377)
(pre-checks) and [moq-dev/moq#7588](https://github.com/moq-dev/moq/pull/7588).
- Build concurrency:
- ✅ 1, maybe 2: start the merge queue at build concurrency 1, and consider
2 only with measured data. At 1, each `merge_group` holds exactly one PR
on top of the base, so a failure blames that PR with no triage.
- Re-enqueue:
- ✅ The back of the queue, so a PR being fixed never jumps PRs that
passed
- The front
- Attempts:
- ✅ Unlimited while progressing, handing back on a round with no new
commit for a failure that isn't flaky, the failure a fix addressed
recurring right after it, or three flaky rounds in a row. This replaces
batch triage's ([moq-dev/quest#355](https://github.com/moq-dev/quest/pull/355))
one attempt per pick; the host gate applies the same rules
([moq-dev/quest#384](https://github.com/moq-dev/quest/pull/384)).
- Flaky failures:
- ✅ The fixer re-enqueues them unchanged, and a test seen flaking twice is
disabled in a separate PR with a follow-up quest; the rerun before blame
(#375) is dropped. Done in [moq-dev/quest#406](https://github.com/moq-dev/quest/pull/406).
Refined 2026-11-03 (coordinator, /quest-complete follow-ups): a failure
that reproduces on the base or comes from the runner is not a flaky
verdict. Done in [moq-dev/quest#468](https://github.com/moq-dev/quest/pull/468).
The end-to-end run covers the first disable PR, and the host gate gets
no flaky record, since its work is frozen until it is retired.
- Pick record (decided 2026-11-02, replacing "Picks GitHub refuses to
enqueue: ✅ The watcher fixes them"):
- ✅ GitHub's records only. Drop the comment and `pick.ts`. The watcher
reads queue events and auto-merge state. A pick GitHub refuses fails the
command, and /quest-merge recommends /quest-iterate as it does today.
Bad picks shrinks to enqueueing PRs with auto-merge armed that GitHub
never queued, which `quest land watch` now does
([community#209348](https://github.com/orgs/community/discussions/209348)).
- Comment only on refusal
- Keep the comment on every pick

Reasons: for queued picks, GitHub already keeps a durable,
writer-authenticated record. `AddedToMergeQueueEvent` and
`RemovedFromMergeQueueEvent` give the enqueuer (who must have write
access), the head before removal, and the reason, and `autoMergeRequest`
gives `enabledBy`/`enabledAt` for picks waiting on checks. Auto-merge can't
reliably be armed on a conflicting PR, and on merge-queue bases enabling it
returns HTTP 422 unless every requirement is met
([community#190610](https://github.com/orgs/community/discussions/190610),
[community#50327](https://github.com/orgs/community/discussions/50327)).
/quest-merge already refuses heads with failing checks or conflicts, so a
refusal only happens if the base moves between that check and the enqueue.
See `quest land watch` (`src/cli/land-watch.ts`, `enqueueArmed`).
- Milestone:
- ✅ m0, at the top

Decided 2026-11-06 (maintainer, /quest-plan), since GitHub offers merge
queues on private repositories only under Enterprise Cloud and moq-dev is on
GitHub Team: moq.pro leaves the host gate for an own serial queue in
`quest land watch`, rather than a plan upgrade, making moq.pro public, or a
third-party queue. The maintainer expects the own queue to become every
repository's, for visibility. The own queue landed in
[moq-dev/quest#540](https://github.com/moq-dev/quest/pull/540). moq.pro's
`main` moved onto it on 2026-11-12; on 2026-11-13 its queue dropped a PR for
a failing build, and the fixer fixed and re-enqueued it.

Decided 2026-11-09 (user, /quest-plan), from a workflow audit of
moq-dev/moq's queue, 2026-11-03 to 11-10: 800 `merge_group` builds at a
5-minute median, 10% failing; the runner 86-90% busy at 19-22 UTC and 53%
overall; enqueue to merge at a 47-minute median; and 245 conflict removals
against 756 merges, 130 of them resolved by the quest merge driver alone.

- Build concurrency, now with the measured data the 2026-11-01 decision
asked for: ✅ stay at 1 (rejected: plan concurrency 2 now).
Re-measure before building for groups of two.
- Re-enqueue, for a conflict the merge driver resolved alone: ✅ keep the
back of the queue for all (rejected: GitHub's `jump` to the front for
those, recommended by the agent for the 95-minute median they wait).

Shared shape, for the children to follow:

- The watcher is a long-running host daemon, one per base branch of a
repository, each with its own clone, locks and state, that reuses
the host gate's fixer infrastructure rather than a new one: moq-bot's App
key through systemd `LoadCredential`, the ephemeral nspawn fixer container
and its launcher, the detached fixer supervisor and its collection in
`src/cli/land-fixer.ts`, `validateFix`'s protected paths, the ready-commit
handback that `quest land apply` rebuilds, and the daemon posting the
comments the fixer writes. `quest land watch` is that daemon; the
batching host gate, `quest land serve`, is retired.
- The fixer prompt (`assets/land-fix.md`) stays as the finished land fixer epic left it.
- The review policy is unchanged: a head the fixer pushed as a recorded
mechanical fix lands without a new review (maintainer, 2026-10-30). A push
by anyone else ends the pick, as does a person removing the PR from the
queue.
- moq's `merge_group` builds may call `quest gate`; the landing priority
class (`QUEST_GATE_PRIORITY=1`, which waits for memory but never for load)
lets them admit ahead of agent load, set from moq's workflow or the
runner's environment.
Every other part of the run is done (checked 2026-11-14, from the watcher's
state on the maintainer's host and moq's PR comments):

- A failing build fixed and re-enqueued:
[moq-dev/moq#8672](https://github.com/moq-dev/moq/pull/8672) (`check
biome`) and [#8722](https://github.com/moq-dev/moq/pull/8722) (`cargo fmt`),
both merged.
- A conflict resolved and re-enqueued:
[moq-dev/moq#8439](https://github.com/moq-dev/moq/pull/8439), once by the
merge driver alone and three times by the fixer, then merged.
- A flaky failure re-enqueued unchanged:
[moq-dev/moq#8468](https://github.com/moq-dev/moq/pull/8468),
[#8551](https://github.com/moq-dev/moq/pull/8551) and
[#8694](https://github.com/moq-dev/moq/pull/8694), all merged.
- A design conflict handed back:
[moq-dev/moq#8104](https://github.com/moq-dev/moq/pull/8104#issuecomment-6704142614),
a modify/delete conflict on a quest file that "requires a content decision".

The watcher's `flaky.json` holds four tests, each with one verdict, so no
disable has run yet. The disable's fixed templates, the follow-up quest and
the epic README, pass `quest check` on moq's `main` as of
[moq-dev/moq#8668](https://github.com/moq-dev/moq/pull/8668).

Once the disable PR exists, check that it landed through the queue with no
maintainer's pick and that moq's `main` passes `quest check`, then delete this
epic. A disable the watcher refuses, or a disable PR GitHub refuses to
enqueue, is a bug in `quest land watch` to fix here.

## Waiting

- moq-dev/moq: one of the four tests in the watcher's `flaky.json` is judged
flaky in a second `merge_group` build, and the watcher opens its disable PR.
Check with `gh pr list -R moq-dev/moq --state all --search
"head:quest-land/flaky-"`.
Loading