ci: cancel obsolete merge queue runs by kumarUjjawal · Pull Request #26760 · apache/datafusion

kumarUjjawal · 2026-10-26T05:23:29Z

Which issue does this PR close?

Related to #25148. Inspired by apache/airflow#75619.

Rationale for this change

Merge queue runs continue using runners after their group is removed or replaced. The Rust workflow gives each merge-group SHA a separate concurrency group, so a replacement does not cancel the old run. Jobs still waiting for a runner can start long after the queue has stopped using their results.

A read-only analysis of runs created on October 19–25, 2026 (UTC) found:

Measurement	Result
Merge-group workflow runs inspected	1,170
Distinct merge-group commits / PRs	195 / 154
Removed or superseded groups with subsequent runner use	36
Workflows / jobs with subsequent runner use	91 / 473
Runner time after removal or replacement	20.3 hours
Jobs that started after removal or replacement	312, using 12.7 runner-hours

For the Rust and extended-test workflows, this was 19.0 of 369.4 runner-hours, or 5.2% of their sampled runner time. Cancellation taking effect one minute after obsolescence would have avoided 17.9 runner-hours; at five minutes, 11.9 hours; at a fixed fifteen-minute delay, 9.4 hours. A separate model using the next exact quarter-hour boundary estimates 11.2 hours.

Examples:

#26562: after manual queue removal, 34 jobs used another 121.6 runner-minutes in parallel. They finished within about 14 minutes. Rust run.
#26373: the group was removed for failed checks at 21:21 UTC on October 22, but its MSRV job did not start until 23:20 UTC. Rust run.

The cutoff was an exact-SHA queue-removal event or the creation of a replacement group for the same PR. For each job, the analysis summed time after the later of its start and the cutoff plus the assumed cancellation delay. It excluded skipped jobs, deduplicated successful jobs copied into rerun records, and excluded seven rerun attempts started after obsolescence that could have been deliberate debugging. All Rust and extended-test jobs were fetched for the denominator; other workflows were inspected when their run metadata showed activity after the cutoff. Merged groups and all rerun attempts contributed zero to the reported avoidable time, so preserving both leaves these estimates unchanged.

These are historical runner times and estimated avoidable portions, not deployed savings or billing estimates. Of the 312 late-starting jobs, 42 used Runs-On (2.4 runner-hours); 270 used GitHub Linux runners. We have not measured whether cancelling queued jobs immediately terminates instances already starting. The models also exclude ref-deletion latency, cancellation cleanup, scheduler delays, and cleanup-workflow overhead. Recent runner changes make this sample unsuitable as an annual forecast.

What changes are included in this PR?

Scan when a new merge group is created and every 15 minutes. Ref deletion can lag the event, so prompt cancellation is not guaranteed.
Ship in observation mode. Cancellation requires the repository Actions variable MERGE_QUEUE_CLEANUP_ENABLED=true. Document at least one week of observation before enabling it; there is no automatic activation.
Log candidate run IDs, SHAs, attempts, reasons, and UTC observation times for comparison with eventual outcomes. Each merge-group scan has its own concurrency group, keyed by the group SHA, so a waiting scan on a group commit is never replaced and shown as cancelled. Scheduled and manual scans keep one group per mode, so a manual dry run cannot replace a waiting cancellation scan.
Target only first-attempt, unfinished merge_group runs in apache/datafusion whose exact queue ref is missing or points elsewhere. Recheck the run and ref, then compare with its target branch to preserve commits already merged. Preserve reruns, other event types, and cleanup itself.
Keep exact-ref detection while the GraphQL queue-entry-to-group-SHA mapping remains unvalidated. A changed ref target is a safeguard for a reused queue branch; the suffix is not treated as the run SHA.
If an Actions run listing reaches the 1,000-result cap, stop the scan without cancelling any runs. Active merge-group runs stay far below the cap.
Share the ref lookup and the target-branch comparison among the workflows of one group.
API errors preserve affected runs and allow other candidates to be checked. Merge-group scans warn; scheduled/manual scans fail visibly. Recheck HTTP 409 responses and retry unresolved cancellations on later scans.
Check out the default branch's script. The workflow definition itself still comes from the triggering revision.
Add behavioral tests to Dev CI and the local lint suite; remove the test that repeated YAML literals.

What is the testing strategy for this PR?

35 cleanup tests cover active/merged groups, missing/replaced refs, target branches containing slashes, reruns, races, dry runs, API errors, error reporting by trigger, pagination, the 1,000-result cap, and shared comparisons. Together with existing fork-CI tests, 59 Node tests pass.
Mutation checks confirm the merged-commit, rerun, listing-cap and shared-comparison regressions are detected.
cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and the complete ./dev/rust_lint.sh suite pass, including generated docs, links, dependency checks, Rust docs, and the HTML documentation build.
actionlint 1.7.12 passes for both changed workflows.
A live read-only probe completed five listing calls with no eligible candidates. No workflow was cancelled during validation. The observation period, production cancellation latency, and instance shutdown behavior remain to be validated after deployment.

Are there any user-facing changes?

Initially the workflow reports candidates only. Once enabled by a maintainer, it cancels obsolete first-attempt merge-queue runs while preserving merged commits and deliberate reruns. Contributors can rerun obsolete jobs for debugging. No query behavior or public API changes.

comphead

Thanks for the analysis. A few design questions before this starts cancelling runs. I'd suggest shipping with cancellation off for a week and comparing the "would cancel" log against the analysis.

Does cancelling a queued job release a runs-on instance that is already starting? That decides how much of the 312 late-starting jobs is real savings.
Filtered run listings cap at 1,000 results.
A manual dry run shares the concurrency group, so it can replace a pending real scan.

comphead · 2026-10-26T16:23:27Z

+
+      const head = await refSha(github, repo, current.head_branch);
+      if (head === current.head_sha) continue;
+      const reason = head === null ? "queue ref removed" : "queue ref replaced";


A merged group also loses its ref. Non-required jobs still running at merge (e.g. cargo check datafusion-ffi features) get cancelled here, and main shows a cancelled run. Could this keep runs already on the base branch, e.g. compareCommitsWithBasehead status identical or behind?

comphead · 2026-10-26T16:23:27Z

+on:
+  # A replacement group starts cleanup promptly; the sweep also catches groups
+  # removed without a replacement and runs that appear after an earlier scan.
+  merge_group:


Does this event arrive before or after the old group's ref is deleted? If before, this scan finds nothing and everything waits for the cron (17.9h vs 9.4h saved by the PR's numbers).

comphead · 2026-10-26T16:23:27Z

+// Called on each new merge group and on a schedule. The queue's exact Git
+// refs, rather than PR state or failing checks, determine which runs are stale.


Why refs rather than GraphQL mergeQueue(branch: "main").entries { headCommit { oid } }? One query gives the active set and removes the per-ref lookups, cache and regex. Also, the ref suffix is the base SHA, so "queue ref replaced" seems to need the same PR re-queued on the same base. Is that the intended case?

comphead · 2026-10-26T16:23:27Z

+concurrency:
+  group: merge-queue-cleanup
+  cancel-in-progress: false


These runs attach checks to group SHAs that become main commits. Pending runs replaced here show as cancelled, and setFailed on API errors shows red. Would warnings be enough?

The setFailed part is fixed, thanks. The cancelled part remains. A queue rebuild creates several groups at once, so one scan runs, the next waits as pending and a third replaces it. The replaced run shows as cancelled on a group commit that can land on main. From Oct 19 to 27, three Rust merge-group runs started within 60 s of each other 15 times. Could merge_group scans use a per-SHA group, or drop that trigger if the observation week shows it rarely finds anything?

comphead · 2026-10-26T16:23:27Z

+        !eligible(current, fullName, context.runId) ||
+        current.head_branch !== candidate.head_branch ||
+        current.head_sha !== candidate.head_sha ||
+        current.run_attempt !== candidate.run_attempt


The next sweep cancels the rerun anyway, so this guard (and the 409 recheck) lasts at most 15 minutes. Exempt run_attempt > 1 for real, or drop the attempt logic?

comphead · 2026-10-26T16:23:28Z

+          // A 409 can mean a completion race, but also an active run that
+          // GitHub cannot cancel. Only suppress it after confirming the race.
+          if (error.status === 409) {


What does cancel return for a run that is already cancelling? If 409, the run is still in_progress and the scan fails every sweep until it finishes. I haven't verified.

comphead · 2026-10-26T16:23:28Z

+          persist-credentials: false
+      - name: Cancel obsolete merge-group runs
+        # During the PR that introduces this workflow, main has no script yet.
+        # Always execute main's script, never code from the merge group.


For merge_group the workflow file comes from the group commit, so a queued PR editing this file runs its own script. Maybe soften to "Execute main's copy of the script".

comphead · 2026-10-26T16:23:28Z

+  assert.equal(new Set(result.cancellations).size, 205);
+});
+
+test("workflow loads default-branch code and supports prompt cleanup, sweeping, and dry runs", () => {


This re-asserts the YAML literals and won't catch a behavior bug. Could it go?

kumarUjjawal · 2026-10-27T04:01:36Z

Thank you @comphead

Pushed new changes to address the comments

comphead

Thanks for the update. Merged commits, reruns, observation mode, the dry-run concurrency split and the merge-group warnings all look good to me.

Optional: compareCommitsWithBasehead runs once per workflow while the ref lookup is cached per branch. Caching the comparison by SHA would save a few calls per group.
Optional: the contributor guide section now reads like a rollout runbook. The observation checklist might fit better in #25148.

comphead · 2026-10-27T15:30:25Z

+
+// Filtered Actions listings are capped at 1,000 results even with pagination.
+// Split full windows by creation time before trusting the result set.
+async function listRuns(github, repo, status, start, end) {


Thanks for handling the cap. Active merge-group runs stay far below 1,000 (167 Rust merge-group runs in all of Oct 19 to 27). Would failing the scan when total_count >= 1000 be enough? It drops the recursive split and its two tests.

comphead · 2026-10-27T15:30:25Z

+      assert.equal(`${owner}/${repo}`, fullName);
+      calls.push(`cancel:${run_id}`);
+      if (cancelErrors[run_id]) throw cancelErrors[run_id];
+      calls.push(`cancel:${run_id}`);


This pushes cancel:${run_id} a second time. Was it meant to record something else?

comphead

Thanks @kumarUjjawal just some nits left

kumarUjjawal · 2026-10-28T04:34:50Z

Thank you @comphead for the review.

Let's test this

ci: cancel obsolete merge queue runs

6ec3259

github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Oct 26, 2026

kumarUjjawal requested review from comphead and neilconway October 26, 2026 05:24

comphead reviewed Oct 26, 2026

View reviewed changes

ci: preserve merged runs and stage queue cleanup rollout

2ef0b61

kumarUjjawal requested a review from comphead October 27, 2026 04:01

comphead reviewed Oct 27, 2026

View reviewed changes

comphead approved these changes Oct 27, 2026

View reviewed changes

kumarUjjawal added 2 commits October 27, 2026 22:33

ci: address merge queue cleanup review nits

aa88eb2

docs: remove rollout plan link from merge queue cleanup section

c39896e

kumarUjjawal mentioned this pull request Oct 27, 2026

Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148

Open

kumarUjjawal added this pull request to the merge queue Oct 28, 2026

Merged via the queue into apache:main with commit d1684ab Oct 28, 2026
47 checks passed

kumarUjjawal deleted the kumar/cancel-obsolete-merge-queue-runs branch October 28, 2026 04:43

github-actions Bot added the v56.0.0 label Oct 28, 2026

		// Called on each new merge group and on a schedule. The queue's exact Git
		// refs, rather than PR state or failing checks, determine which runs are stale.

Conversation

kumarUjjawal commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumarUjjawal commented Oct 27, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

kumarUjjawal commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kumarUjjawal commented Oct 26, 2026 •

edited

Loading