docs: how to check the scheduled CI runs are actually running

[andygrove]

Which issue does this PR close?

Relates to #5999. It does not close it: that issue is about fixing the Miri job, this documents how
to notice a job in that state in the first place.

• Rendered page — CI guide
• Rendered page — release process

Rationale for this change

After #5939 and #5963, most of our coverage of the non-default Spark and Iceberg versions runs in
the nightly tier rather than on pull requests. That was the right trade for cost, but it moves a
lot of signal onto a schedule, and a scheduled run has no pull request to turn red. ci.yml has a
nightly_report job that opens a ci-nightly-failure issue, so that tier is covered. miri.yml
and publish_snapshot.yml have no such step — and both are currently broken without anyone having
noticed. Miri has failed 79 nights running (#5999) and the nightly SNAPSHOT publish has never
succeeded since it landed in #5902, failing with a 401 against repository.apache.org.

There is also a subtler failure I could not find documented anywhere. For ci.yml, a green
Required Checks does not mean the nightly suites ran. The path filters and the diff base are both
allowed to select nothing, which is correct on a documentation-only day, but a run that tested
nothing is indistinguishable from a run that tested everything if you only look at the verdict. If
the diff base ever drifts, the nightly goes green forever while testing nothing.

I checked the nightly tier itself and it is working — the 2026-09-16 and 2026-09-17 runs each
completed about 40 Spark SQL and Iceberg suite jobs, with shards running 25 to 67 minutes, and the
second run correctly resolved its diff base to the first run's head. So there is nothing to fix in
the tier.

What changes are included in this PR?

A new "Checking that the scheduled runs are healthy" section in the CI contributor guide, listing
the four schedules and which of them report their own failures, with a gh api loop over the three
daily ones and guidance on reading the output for the two distinct failures — a schedule that
stopped firing versus a streak of red nights. It also gives a second command that counts the
nightly's suite jobs by conclusion, so "green because everything passed" can be told apart from
"green because nothing ran."

The release process gets a checklist item and a paragraph under Release Preparation pointing at
that section, so this gets looked at once per release rather than only when someone happens to
wonder.

The documented commands use gh api rather than gh run list --event=schedule. The latter is not
reliable: while investigating this it omitted that morning's ci.yml scheduled run entirely, and
on another call returned three-week-old miri.yml runs as the latest three.

Following review, the guide also opens with a diagram of how the suites split across the pull
request, merge queue and nightly tiers, ahead of the table that has the detail. That is the one
change here that is not purely documentation: the site is built with Sphinx, where a bare
```mermaid fence renders as a literal code block, so sphinxcontrib-mermaid goes into
docs/requirements.txt and myst_fence_as_directive into conf.py. One fence then renders as a
diagram both on the published site and on github.com. Worth flagging that the extension draws the
diagram in the browser from cdn.jsdelivr.net, which would be the first third-party asset the
Comet site loads — say the word and I will drop the diagram rather than introduce that.

How are these changes tested?

The Sphinx change is the only part with a build behind it, and the docs job is push-only, so
nothing on this pull request exercises it. I built it locally instead: installing
docs/requirements.txt and running a Sphinx HTML build emits <pre class="mermaid"> for the fence
rather than a highlighted code block, and injects mermaid@11.12.1 from jsdelivr. The diagram
source parses clean against that same mermaid version, checked alongside a deliberately broken
control to confirm the check actually rejects. myst_fence_as_directive exists in myst-parser 2.0,
which is the floor already pinned in docs/requirements.txt.

Every command in the new section was run against the live repository and produced the output the
surrounding text describes. The gh run view one was re-run with GH_REPO pointed at a fork, to
confirm it 404s without --repo and returns 40 successes and five skips with it.
prettier --check "**/*.md" and python3 dev/ci/check-ci-config.py both pass.

[comphead]

Thanks @andygrove
would be nice to include a mermaid diagram to how CI is split between PR level, MQ level, nightly

[sunchao]

Correctness

The existing guide explains CI tiers and failure triage but lacks a routine check that scheduled coverage is still arriving. This adds the daily history query, a job-conclusion query, and a release-preparation reminder. I reviewed the two authored documentation files at 2a03ec94 against base 58ab5f61, separating the upstream changes since the branch point. Spark execution and expression semantics are unaffected by this documentation scope.

The four listed CI schedules match their workflow definitions. ci.yml selects the scheduled event separately from PR and manual runs, routes changes through the nightly policy, and gives the failure reporter actions: read and issues: write. The stale-date and enabled-state checks make a missing daily schedule observable. GitHub documents the default-branch restriction and possible delayed, dropped or inactivity-disabled schedules in its schedule reference.

One P2 remains inline: the job query obtains a run ID from upstream but can look it up in the caller's selected fork. I reproduced the 404 and confirmed that adding --repo apache/datafusion-comet fixes it. Against upstream, the exact documented commands work. The latest scheduled run is on main, and the job query returns 40 successes and five skips. Its Detect changes log records the prior successful nightly's base, so the proposed investigation path is usable.

The current PR checks have seven successes and 15 skips, with no failures. Preflight checked merge 1f72a376, whose parents are the pinned base and head. Its log confirms Prettier and CI configuration checks passed. Both edited files and the relevant workflow definitions match the reviewed sources. Validation here used read-only commands and source comparison.

Performance

The change adds documentation and on-demand reads of recent run metadata. It introduces no scheduled job, build work or runtime allocation. The five-run limit keeps the history query bounded, while inspecting job conclusions avoids downloading test logs just to establish whether coverage ran. There is no new performance claim requiring a benchmark.

Design

Separating schedule freshness, repeated failures and actual suite execution addresses distinct ways a nightly check can mislead a release manager. Linking the release checklist to one operational guide keeps that process discoverable. Explicitly selecting upstream in the second command is the concrete change needed to make the sequence work consistently across contributor checkouts.

Abstraction & complexity

The implementation stays within two existing Markdown pages and uses the existing GitHub API and CLI. The release page points to the detailed procedure rather than maintaining another copy of the commands. No new helper, configuration layer or workflow indirection is introduced.

[sunchao]

Correctness

[P2] Pin the job lookup to the same repository as the run lookup

Could this call include --repo apache/datafusion-comet? The preceding gh api always obtains an upstream run ID, but gh run view uses the locally selected repository. With GH_REPO=andygrove/datafusion-comet, the documented lookup obtains upstream run 35189906693 and then fails with HTTP 404 at /repos/andygrove/datafusion-comet/actions/runs/35189906693. Adding --repo apache/datafusion-comet under that same environment succeeds and returns five skipped and 40 successful jobs. Contributors working with a fork selected cannot perform the documented coverage check as written. An explicit repository keeps this command consistent with the other checks and removes its dependency on the caller's checkout.

[andygrove]

Good catch, thanks — fixed in bacd419. You are right that the two halves of that snippet disagree with each other: the run ID is pinned to upstream and the lookup that consumes it is not, so it only works if your checkout happens to have upstream selected. I reproduced the 404 the same way, and with --repo apache/datafusion-comet added it returns the 40 successes and five skips the surrounding text describes.

[andygrove]

@comphead added one in bacd419, at the top of the "Three tiers" section so it sets up the table rather than repeating it. It shows the four events down the left, the tier each one runs, and the label opt-in as dotted edges, including Spark 3.4 sitting on its own with nothing but a label pointing at it.

One thing I would like your read on before this goes in. Our site is Sphinx, and a plain ```mermaid fence renders there as a literal code block, so making the diagram actually draw meant adding sphinxcontrib-mermaid to `docs/requirements.txt` and `myst_fence_as_directive` to `conf.py`. That builds clean locally and the fence comes out as a mermaid element rather than a highlighted block, but the `docs` job is push-only, so nothing on this PR exercises it — it would first run after merge.

The part I am less sure about is that the extension draws the diagram in the browser and pulls mermaid.js from cdn.jsdelivr.net, which as far as I can tell would be the first third-party asset the Comet site loads. I do not think that is a problem, but it is a change in kind rather than degree for an ASF site, so I would rather you agreed than assume. If you would rather not take it on, I am happy to drop the diagram and keep this documentation-only.

[andygrove]

Correcting myself on the comment above: I built the site locally to look at the diagram and found that jsdelivr would not be the first third-party asset the Comet site loads. docs/source/conf.py already pulls buttons.github.io/buttons.js, and docs/source/_templates/layout.html pulls Inter and JetBrains Mono from fonts.googleapis.com and fonts.gstatic.com. Both are ours, not the theme's.

So this is a fourth external host rather than a change in kind, which makes the concern I raised weaker than I put it. Still worth you nodding at, but it is not the precedent I implied.

[andygrove]

[sunchao]

Re-reviewed bacd4199 against 58ab5f61, including the update since 2a03ec94. The P2 repository-selection issue is fixed. From outside the clone with GH_REPO=andygrove/datafusion-comet, the original lookup still returns 404, while the revised --repo apache/datafusion-comet command returns 40 successful and five skipped jobs for the same upstream run. No verified P1/P2 findings remain.

The new diagram matches the CI routing policy. A local Sphinx build with the declared requirements succeeds and emits the diagram as Mermaid markup with the versioned JavaScript import. The release-page link resolves. This was a direct source build, with 61 warnings about missing generated pages or cross-references outside the changed pages. I did not run the full release-doc generation or browser JavaScript rendering.

At 2026-09-17 23:36:19 UTC, seven checks passed and 15 were skipped, including the docs deployment. Preflight ran on merge a3c3d2a with parent d0c10f2d, rather than the assigned base. All four authored files match that merge. Markdown formatting passed there, and the exact-head CI configuration check also passed locally.

[andygrove]

Thanks @andygrove would be nice to include a mermaid diagram to how CI is split between PR level, MQ level, nightly

Added