From 2a03ec94b69e55e2b15f1e726720591a75174e17 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 17 Sep 2026 09:02:46 -0600 Subject: [PATCH 1/2] docs: how to check the scheduled CI runs are actually running Most of Comet's coverage of the non-default Spark and Iceberg versions moved to the nightly tier in #5963, and a scheduled run has no pull request to turn red. `ci.yml` opens a `ci-nightly-failure` issue when it goes red, but `miri.yml` and `publish_snapshot.yml` report nothing at all, which is how Miri came to fail 79 nights in a row unnoticed (issue #5999). Add a section to the CI guide covering the two distinct failures: a schedule that stopped firing, and a streak of red nights. For `ci.yml` specifically, green does not mean the suites ran -- a quiet day and a drifted diff base both report the same green `Required Checks` -- so include a second command that counts suite jobs by conclusion to tell them apart. Hook it into the release process as a preparation step, so a release does not go out on the assumption that the nightly tier has been covering the versions the tier table says it covers. Use `gh api` rather than `gh run list --event=schedule` in the documented commands: the latter returned three-week-old runs as the latest three while investigating this. --- docs/source/contributor-guide/ci.md | 52 +++++++++++++++++++ .../contributor-guide/release_process.md | 9 ++++ 2 files changed, 61 insertions(+) diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 20f2eedf808..9d252fd0fd3 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -244,6 +244,58 @@ stays in one place. When you pick up a nightly failure: Dispatching `ci.yml` from the Actions page with **Run workflow** runs every tier, including the nightly suites, if you need a result before the next scheduled run. +## Checking that the scheduled runs are healthy + +A scheduled run has no pull request to turn red, so when one breaks, nothing puts it in front of +anyone. Comet has three daily schedules plus a weekly one, and two of the daily ones report nothing +when they fail: + +| Workflow | Cron (UTC) | Reports a failure? | +| ---------------------- | ------------ | ------------------------------------- | +| `publish_snapshot.yml` | `0 3 * * *` | no | +| `miri.yml` | `0 4 * * *` | no | +| `ci.yml` nightly tier | `0 6 * * *` | yes, a `ci-nightly-failure` issue | +| `codeql.yml` | `16 4 * * 1` | yes, to the repository's Security tab | + +So the two without a reporting step have to be looked at deliberately. Check all of them at once: + +```sh +for wf in ci.yml miri.yml publish_snapshot.yml; do + gh api "repos/apache/datafusion-comet/actions/workflows/$wf/runs?event=schedule&per_page=5" \ + --jq ".workflow_runs[] | \"$wf\t\(.created_at[0:10])\t\(.conclusion)\t\(.html_url)\"" +done +``` + +Read the output for two different things: + +- **A missing row.** If the most recent run is not from the last day or two, the schedule itself + stopped firing. GitHub drops scheduled runs under load and disables them entirely in a repository + with no activity for 60 days, and it does not announce either. Confirm the workflow is still + enabled with `gh api repos/apache/datafusion-comet/actions/workflows --jq '.workflows[] | "\(.state)\t\(.path)"'`, + and re-enable it from the Actions page if it is `disabled_inactivity`. +- **A run of failures.** One red night is a flake or a real regression, and for `ci.yml` there is an + issue open about it. Several consecutive red nights on `miri.yml` or `publish_snapshot.yml` means + nobody has looked; treat the streak, not the newest run, as the thing to explain. + +For the `ci.yml` nightly, green on its own does not mean the suites ran. Path filters and the diff +base are both allowed to select nothing — a documentation-only day legitimately runs no suite at +all — and a run that tested nothing reports exactly the same green `Required Checks` as a run that +tested everything. Confirm the suites actually ran: + +```sh +run=$(gh api "repos/apache/datafusion-comet/actions/workflows/ci.yml/runs?event=schedule&per_page=1" \ + --jq '.workflow_runs[0].id') +gh run view "$run" --json jobs \ + --jq '[.jobs[] | select(.name | test("^(Spark SQL|Iceberg Spark SQL) Tests"))] + | group_by(.conclusion)[] | "\(length)\t\(.[0].conclusion)"' +``` + +A healthy run over a day of normal merges reports about 40 successes, with the handful of skips +being the suites that belong to the queue tier rather than the nightly one — Spark 3.4, Spark 4.1 +and Iceberg 1.11. If everything is skipped, open the run's `Detect changes` job: it logs the +`Nightly base:` commit it diffed against and the list of changed files, which is enough to tell a +genuinely quiet day from a base that has drifted. + ## Reproducing a suite failure locally The Spark SQL suites outside the PR tier run Spark's own test suite against Comet, with the diff --git a/docs/source/contributor-guide/release_process.md b/docs/source/contributor-guide/release_process.md index 2cc430a1483..ae364b3692d 100644 --- a/docs/source/contributor-guide/release_process.md +++ b/docs/source/contributor-guide/release_process.md @@ -29,6 +29,7 @@ The following is a quick-reference checklist for the full release process. See t instructions on each step. - [ ] Release preparation: review expression support status and user guide +- [ ] Release preparation: check the scheduled CI runs are healthy - [ ] Create release branch - [ ] Protect the release branch in `.asf.yaml` - [ ] Generate release documentation @@ -70,6 +71,14 @@ It is also recommended to run benchmarks (such as TPC-H and TPC-DS) comparing pe release to check for regressions. See the [Comet Benchmarking Guide](benchmarking.md) for instructions. +Check that the scheduled CI runs have been healthy over the release window. Most of Comet's coverage of the +non-default Spark and Iceberg versions runs nightly rather than on pull requests, so a nightly run that has been +failing — or one that silently stopped firing — means the release is going out with less testing behind it than the +tier table suggests. A scheduled run has no pull request to turn red, and `miri.yml` and `publish_snapshot.yml` do +not report their own failures at all, so this has to be looked at deliberately. See +[Checking that the scheduled runs are healthy](ci.md#checking-that-the-scheduled-runs-are-healthy) for the commands +and for how to tell a genuinely quiet night from a broken one. + These are tasks where agentic coding tools can be particularly helpful — for example, scanning the codebase for newly registered expressions and cross-referencing them against the documented list, or generating test queries to verify expression support status. From bacd419987ec421a272a93e8c6657ffe07b9092b Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 17 Sep 2026 16:13:57 -0600 Subject: [PATCH 2/2] docs: add a CI tier diagram, pin the job lookup to the upstream repo Review feedback on #6000. comphead asked for a diagram of how CI splits across the PR, merge queue and nightly tiers. The docs site is Sphinx and had no mermaid support, so a bare ```mermaid fence would have rendered as a literal code block there while rendering as a diagram on github.com. Add sphinxcontrib-mermaid and set myst_fence_as_directive so the one fence renders as a diagram in both places. The extension loads mermaid.js from cdn.jsdelivr.net at the version it pins. sunchao found that the job-conclusion command mixes repositories: the run ID comes from an explicit `gh api` call against apache/datafusion-comet, but the `gh run view` that consumes it resolves against whatever repository the caller has selected, so it 404s for anyone working from a fork. Pass --repo. --- docs/requirements.txt | 1 + docs/source/conf.py | 9 +++++++++ docs/source/contributor-guide/ci.md | 22 +++++++++++++++++++++- 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index 80505f2aa53..10eb4f693d1 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -20,5 +20,6 @@ sphinx>=7.0,<8.0 sphinx-reredirects pydata-sphinx-theme>=0.16.1,<0.17.0 myst-parser>=2.0,<4.0 +sphinxcontrib-mermaid>=1.0,<3.0 maturin jinja2 diff --git a/docs/source/conf.py b/docs/source/conf.py index 63f8f0114b5..ba325734e02 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -53,6 +53,10 @@ 'sphinx.ext.napoleon', 'myst_parser', 'sphinx_reredirects', + # Renders the ```mermaid fences in the contributor guide. The diagrams are drawn in + # the browser by mermaid.js, which sphinxcontrib-mermaid loads from cdn.jsdelivr.net + # at the version pinned by its `mermaid_version` default. + 'sphinxcontrib.mermaid', ] source_suffix = { @@ -60,6 +64,11 @@ '.md': 'markdown', } +# Route ```mermaid fences to the mermaid directive rather than treating them as a literal +# code block, so the same source renders as a diagram here and on github.com, which +# understands that fence natively. +myst_fence_as_directive = ['mermaid'] + # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 9d252fd0fd3..585d0c589eb 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -32,6 +32,26 @@ preflight checks first (license headers, Markdown formatting, workflow linting, checks), computes which heavy jobs the changed files are relevant to, and fans out to those jobs. Which jobs run also depends on the event: +```mermaid +flowchart LR + PR([pull request]) --> PRTIER + PR -. with label .-> QUEUE + PR -. with label .-> NIGHTLY + PR -. with label .-> S34 + MQ([merge queue]) --> PRTIER + MQ --> QUEUE + CRON([schedule
06:00 UTC]) --> NIGHTLY + PUSH([push to main]) --> CACHE + + PRTIER["PR tier
Linux build, lint, Rust tests
TPC-H / TPC-DS
Comet suites, Spark 4.1"] + QUEUE["Queue tier, on top of the PR tier
Spark SQL, Spark 4.1
Iceberg 1.11
macOS build and Comet suites
Benchmark check, Delta gate
PyArrow UDF, Spark 4.0 / 4.1 / 4.2"] + NIGHTLY["Nightly tier
Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2
Spark SQL, Spark 3.5 / 4.0
Iceberg 1.8 / 1.9 / 1.10"] + S34["Neither tier
Spark SQL, Spark 3.4"] + CACHE["Cache-refresh-only mode
the four cache-writing jobs, plus Lint"] +``` + +Suite by suite: + | Suite | Pull request | Merge queue | Nightly | | ------------------------------------------------- | ------------ | ----------- | ------- | | Linux build, lint, Rust tests, TPC-H/TPC-DS | yes | yes | no | @@ -285,7 +305,7 @@ tested everything. Confirm the suites actually ran: ```sh run=$(gh api "repos/apache/datafusion-comet/actions/workflows/ci.yml/runs?event=schedule&per_page=1" \ --jq '.workflow_runs[0].id') -gh run view "$run" --json jobs \ +gh run view "$run" --repo apache/datafusion-comet --json jobs \ --jq '[.jobs[] | select(.name | test("^(Spark SQL|Iceberg Spark SQL) Tests"))] | group_by(.conclusion)[] | "\(length)\t\(.[0].conclusion)"' ```