diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 93e2b0374d..0f1f93fe94 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -12,23 +12,28 @@ then fans out to the long-running test/build workflows. Each long workflow is a `workflow_call` reusable invoked from the umbrella. Merging goes through GitHub's merge queue, configured by the `Merge Queue` -ruleset in `.asf.yaml`. That splits CI into two tiers: +ruleset in `.asf.yaml`. That splits CI into three tiers: - **PR tier** (`pr`): fast feedback while a change is being iterated on. The Linux build, with the Comet test suites run against the default Spark profile (4.1) only, and nothing else. -- **Queue tier** (`queue`): the authoritative gate. The Linux build with the - Comet test suites against every Spark profile, plus the macOS build, the - benchmark compile check, the Delta contrib build gate, the PyArrow UDF - suite, Spark 3.5/4.0/4.1 and Iceberg 1.8/1.9/1.10/1.11, - evaluated against the merge result rather than against the PR head. - -Every queue-only job has a `run-*` label that opts a pull request into it -early, listed in the diagram below. The Lint Java matrix compiles Spark -3.4/3.5/4.0 on every pull request, so a shim that fails to build is caught -there; only the runtime suites wait for the queue. - -`spark_3_4` is in neither tier. Spark 3.4 is deprecated, so its Spark SQL +- **Queue tier** (`queue`): the authoritative gate. The PR tier plus the + macOS build, the benchmark compile check, the Delta contrib build gate, the + PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg 1.11, evaluated + against the merge result rather than against the PR head. One Spark version + and one Iceberg version, both the default profile's. +- **Nightly tier** (`nightly`): the regression sweep of everything else, once + a day against `main` as it stands. The Comet test suites against the other + four Spark profiles, Spark SQL on Spark 3.5 and 4.0, and Iceberg + 1.8/1.9/1.10. See [Nightly tier](#nightly-tier) below for how a failure + surfaces. + +Every queue-only and nightly job has a `run-*` label that opts a pull request +into it early, listed in the diagram below. The Lint Java matrix compiles +Spark 3.4/3.5/4.0 on every pull request, so a shim that fails to build is +caught there; only the runtime suites wait for the queue or the nightly. + +`spark_3_4` is in none of the tiers. Spark 3.4 is deprecated, so its Spark SQL suite no longer gates a merge; it runs only when a pull request carries `run-spark-3.4-tests`, or from a `workflow_dispatch`. Anyone who wants to check a change against 3.4 can still do so, but note when that result starts @@ -67,11 +72,11 @@ The profile rows of the `linux-test` matrix live in job-level `if:` cannot see `matrix`: the `lint` job runs the script with the `profiles` input and publishes the rows as a job output that the matrix reads with `fromJSON`. Each row carries a tier, `pr` for the default build profile -and `queue` for the other four, and `check-ci-config.py` asserts that the two +and `nightly` for the other four, and `check-ci-config.py` asserts that the two tiers partition the list and that the `pr` tier is exactly the default profile. ``` - pull_request | merge_group | push to main | workflow_dispatch + pull_request | merge_group | push to main | schedule | workflow_dispatch | v +-----------------------+ @@ -98,16 +103,19 @@ tiers partition the list and that the `pr` tier is exactly the default profile. (Spark 4.1 profile only) pr_benchmark_check run-benchmark-check delta_build_gate run-delta-build-gate pyarrow_udf_test run-pyarrow-udf-tests - pr_build_linux run-all-spark-profiles - (other profiles) - spark_3_5 run-spark-3.5-tests - spark_4_0 run-spark-4.0-tests spark_4_1 run-spark-4.1-tests label or dispatch only spark_4_1 sql_hive run-spark-4.1-hive-tests - ---------------------- iceberg_1_8 run-iceberg-tests - spark_3_4 run-spark-3.4-tests iceberg_1_9 run-iceberg-tests - iceberg_1_10 run-iceberg-tests - iceberg_1_11 run-iceberg-tests + ---------------------- iceberg_1_11 run-iceberg-tests + spark_3_4 run-spark-3.4-tests + nightly tier, or PR with label + ------------------------------ + pr_build_linux run-all-spark-profiles + (other profiles) + spark_3_5 run-spark-3.5-tests + spark_4_0 run-spark-4.0-tests + iceberg_1_8 run-iceberg-tests + iceberg_1_9 run-iceberg-tests + iceberg_1_10 run-iceberg-tests | | | +-----------------------------------+-----------------------------------+ @@ -116,6 +124,14 @@ tiers partition the list and that the `pr` tier is exactly the default profile. | required_checks | ubuntu-slim | one flat name that | | is safe to require | + +-----------+-----------+ + | schedule only, when red + v + +-----------------------+ + | nightly_report | ubuntu-slim + | opens or comments on | + | a ci-nightly-failure | + | issue | +-----------------------+ reusable workflows invoked via `uses:`: @@ -129,23 +145,24 @@ tiers partition the list and that the `pr` tier is exactly the default profile. | Job in `ci.yml` | Triggered by | Routing rule | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- | -| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) | -| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` | -| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full`; the test matrix's non-default Spark profiles only in the merge group **or** with `run-all-spark-profiles`, via `build_linux_all_profiles` | `dev/ci/compute-changes.py` | +| `preflight` | every PR / merge group / push / schedule / dispatch / label | none (always runs) | +| `changes` | every PR / merge group / push / schedule / dispatch / label | runs `dev/ci/compute-changes.py` | +| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full`; the test matrix's non-default Spark profiles only in the nightly run **or** with `run-all-spark-profiles`, via `build_linux_all_profiles` | `dev/ci/compute-changes.py` | | `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` | | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only | | `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` | | `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code | | `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | -| `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | +| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | | `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources | | `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources | -| `spark_4_0` | merge group, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | +| `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | | `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_8` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_9` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_10` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `required_checks` | always, after every job above except `docs` | none (always runs) | +| `nightly_report` | schedule only, after `required_checks`, when it is not green | none | A heavy job appears in the PR's checks list as a `skipped` entry whenever its path filter or event criteria don't match. Skipped checks count as @@ -192,6 +209,40 @@ The opt-in labels have to exist in repository settings before they can be applied; `contains()` on a label nobody can add is simply always false, which makes the escape hatch look like it silently does nothing. +### Nightly tier + +`ci.yml` also fires on a `schedule` (06:00 UTC daily). On that event `changes` +diffs `main` against the commit the last successful scheduled run tested +(`dev/ci/nightly-base.py` looks it up through the Actions API), so the nightly +is routed by the same `FILTERS` as every other event and `POLICY` selects the +`nightly` tier alone: the queue already ran every `queue` job against the tree +that is now `main`. Every commit is covered exactly once, and a red nightly +keeps its commits in scope until a green one supersedes it. A quiet day diffs +to nothing and runs nothing; a docs-only day runs nothing either. + +When there is no such run, the API is unreachable, or the base is no longer on +`main`, there is no base to diff against and the run treats every tracked file +as changed, so the whole nightly tier runs. Guessing a narrower base — a fixed +time window, say — would be worse than useless: a window that starts after a +commit no nightly has covered yet skips the suites that commit needs, lets the +run go green, and then hands that green head to `nightly-base.py` as tomorrow's +base, so the coverage is dropped for good. + +With `profiles: nightly`, `pr_build_linux.yml` runs only `lint`, `build-native` +and the `linux-test` matrix; the lints, Rust tests, Spark build and TPC runs +carry `if: ${{ inputs.profiles != 'nightly' }}` because the queue and the push +run already produced those verdicts at the same commit. `check-ci-config.py` +pins that the same way it pins the cache-refresh guard. + +A red nightly has no pull request to appear on, and GitHub only emails a +scheduled run's failure to whoever last touched the workflow file, so +`nightly_report` runs after `required_checks` on the scheduled event and, when +the aggregate is not green, opens an issue labelled `ci-nightly-failure` listing +the jobs that failed. If one is already open it comments there instead, so +consecutive red nights accumulate in one issue. Closing the issue is how the +failure is acknowledged; the next red night opens a new one. The label has to +exist in repository settings, like the `run-*` labels above. + ## Standalone workflows (not under the umbrella) These workflows have their own triggers because they fire on events the @@ -236,11 +287,15 @@ in `dev/ci/compute-changes.py`: dorny/picomatch (`**` spans path segments, `*` stays within one, a leading `!` excludes). - **`POLICY`** — which events may run it. `"pr"` for every pull request, - `"queue"` for the merge queue, `"push"` for push to main, `"label:"` - for opt-in on a labelled pull request. `"pr"` and `"label:"` are mutually - exclusive. `workflow_dispatch` always runs everything. - -Moving a suite between the PR and queue tiers is a one-word edit to `POLICY`. + `"queue"` for the merge queue, `"nightly"` for the scheduled run, `"push"` + for push to main, `"label:"` for opt-in on a labelled pull request. + `"pr"` and `"label:"` are mutually exclusive. `workflow_dispatch` always + runs everything. + +Moving a suite between the PR, queue and nightly tiers is a one-word edit to +`POLICY`, plus the matching entry in `PR_TIER`, `QUEUE_TIER` or `NIGHTLY_TIER` +in `dev/ci/check-ci-config.py`, which is written out longhand on purpose so +that a tier change has to be stated twice. An output does not have to map one-to-one onto a job. Two outputs can feed a single call when part of a workflow belongs in a different tier from the rest: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9a72a3f21b..d5bdde5aa6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,19 +20,25 @@ # change touched files relevant to that workflow. # # Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so -# there are two tiers: +# there are three tiers: # # pull_request fast feedback. The Linux build only, with its test matrix # run against the default Spark profile (4.1) alone. -# merge_group the authoritative gate. The Linux build with the test -# matrix against every Spark profile, plus the macOS build, +# merge_group the authoritative gate. The PR tier plus the macOS build, # the benchmark compile check, the Delta contrib build gate, -# the PyArrow UDF suite, Spark 3.5/4.0/4.1 and Iceberg -# 1.8/1.9/1.10/1.11, evaluated against the merge result -# rather than the PR head. +# the PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg +# 1.11, evaluated against the merge result rather than the +# PR head. One Spark version and one Iceberg version, both +# the default profile's. +# schedule the nightly regression sweep of everything else: the Linux +# test matrix against the other Spark profiles, Spark SQL on +# Spark 3.5 and 4.0, and Iceberg 1.8/1.9/1.10, routed by the +# same path filters over what landed on main since the last +# green nightly. A failure opens (or comments on) an issue +# labelled `ci-nightly-failure`. # -# Spark 3.4 is deprecated and sits outside both tiers: it runs only when a -# pull request carries `run-spark-3.4-tests`, or from a manual dispatch. +# Spark 3.4 is deprecated and sits outside all three tiers: it runs only when +# a pull request carries `run-spark-3.4-tests`, or from a manual dispatch. # # Which tier a job sits in is POLICY in dev/ci/compute-changes.py, not an # expression here. Heavy jobs deliberately have no `push` tier: the queue @@ -49,10 +55,12 @@ name: CI # A `labeled` event (e.g. the run-spark-*-tests gates, or dependabot's automatic # `dependencies` label added ~1s after open) fires at the same commit as the # opened/synchronize run. Keying the group on the label name keeps labeled runs -# in their own subgroup so they never cancel the real commit run; opened and -# synchronize both map to `commit` so a new push still supersedes its predecessor. +# in their own subgroup so they never cancel the real commit run. Every other +# run is keyed on its event name: opened, synchronize and reopened all map to +# `pull_request`, so a new push still supersedes its predecessor, while a +# scheduled run at the tip of main never cancels that commit's push run. concurrency: - group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name || 'commit' }} + group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name || github.event_name }} cancel-in-progress: true on: @@ -62,6 +70,11 @@ on: push: branches: - main + schedule: + # 06:00 UTC daily: after the evening's merges in the Americas and before + # the working day starts in Europe, so a red nightly is waiting when the + # first people look. Miri runs at 04:00 and the snapshot publish at 03:00. + - cron: '0 6 * * *' workflow_dispatch: jobs: @@ -159,6 +172,11 @@ jobs: name: Detect changes needs: preflight runs-on: ubuntu-slim + # `actions: read` is for dev/ci/nightly-base.py, which lists previous + # scheduled runs to pick the nightly's diff base. + permissions: + actions: read + contents: read outputs: build_linux: ${{ steps.compute.outputs.build_linux }} build_linux_full: ${{ steps.compute.outputs.build_linux_full }} @@ -198,6 +216,7 @@ jobs: MQ_HEAD_SHA: ${{ github.event.merge_group.head_sha }} PUSH_BEFORE: ${{ github.event.before }} PUSH_AFTER: ${{ github.sha }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail : > changed_files.txt @@ -206,6 +225,29 @@ jobs: # every output true for this event so a manual run can exercise # any gated job. : + elif [[ "$EVENT_NAME" == "schedule" ]]; then + # The nightly's base is the commit the last successful scheduled + # run tested, so everything that landed since is covered exactly + # once and a red nightly keeps its commits in scope until a green + # one supersedes it. A quiet day diffs to nothing and runs + # nothing; so does a docs-only day. + # + # Without a base -- the first nightly, an API error, or a base no + # longer on main -- run the whole tier by treating every tracked + # file as changed, the same way the first push to a branch does + # below. Guessing a narrower base would be worse than running + # nothing: a window that starts after a commit no nightly has + # covered yet skips the suites that commit needs, goes green + # anyway, and then becomes the base for tomorrow, dropping that + # coverage for good. + prev=$(python3 dev/ci/nightly-base.py) + if [[ -n "$prev" ]] && git merge-base --is-ancestor "$prev" HEAD 2>/dev/null; then + echo "Nightly base: $prev" + git diff --name-only "$prev"..HEAD > changed_files.txt + else + echo "Nightly base unknown; running the whole nightly tier" + git ls-tree -r --name-only HEAD > changed_files.txt + fi elif [[ "$EVENT_NAME" == "pull_request" ]]; then git diff --name-only "$PR_BASE_SHA"..."$PR_HEAD_SHA" > changed_files.txt elif [[ "$EVENT_NAME" == "merge_group" ]]; then @@ -248,21 +290,22 @@ jobs: # # pull request linux, full -> profiles: pr # ... with the label linux, full, all -> profiles: all - # `labeled` run all -> profiles: queue-only - # merge queue linux, full, all -> profiles: all + # `labeled` run all -> profiles: nightly + # merge queue linux, full -> profiles: pr + # nightly all -> profiles: nightly # push to main linux -> cache-refresh-only # # Only push to main sets `build_linux` without `build_linux_full`, which is # the whole point: the queue has already tested that tree, so the push run - # is there for the caches alone. A `labeled` run sets only the third, and - # then runs just the profiles the PR tier skipped. + # is there for the caches alone. A `labeled` run and the nightly set only + # the third, and then run just the profiles the PR and queue tiers skip. if: needs.changes.outputs.build_linux == 'true' || needs.changes.outputs.build_linux_all_profiles == 'true' uses: ./.github/workflows/pr_build_linux.yml with: cache-refresh-only: ${{ needs.changes.outputs.build_linux_full != 'true' && needs.changes.outputs.build_linux_all_profiles != 'true' }} profiles: >- ${{ needs.changes.outputs.build_linux_all_profiles != 'true' && 'pr' - || needs.changes.outputs.build_linux_full != 'true' && 'queue-only' + || needs.changes.outputs.build_linux_full != 'true' && 'nightly' || 'all' }} pr_build_macos: @@ -332,7 +375,7 @@ jobs: spark_3_5: name: Spark SQL Tests (Spark 3.5) needs: changes - # Queue-only by default; PRs need the `run-spark-3.5-tests` label. + # Nightly by default; PRs need the `run-spark-3.5-tests` label. if: needs.changes.outputs.spark_3_5 == 'true' uses: ./.github/workflows/spark_sql_test_reusable.yml with: @@ -343,9 +386,8 @@ jobs: spark_4_0: name: Spark SQL Tests (Spark 4.0) needs: changes - # Queue-only by default; PRs need the `run-spark-4.0-tests` label. Swapped - # with spark_4_1 on the `oom` branch to validate the memory caps against - # Spark 4.1 by default. + # Nightly by default; PRs need the `run-spark-4.0-tests` label. Spark 4.1 + # is the one Spark SQL suite the queue runs, being the default profile. if: needs.changes.outputs.spark_4_0 == 'true' uses: ./.github/workflows/spark_sql_test_reusable.yml with: @@ -376,7 +418,7 @@ jobs: iceberg_1_8: name: Iceberg Spark SQL Tests (Iceberg 1.8) needs: changes - # Queue-only by default; PRs need the `run-iceberg-tests` label. + # Nightly by default; PRs need the `run-iceberg-tests` label. if: needs.changes.outputs.iceberg_1_8 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: @@ -389,7 +431,7 @@ jobs: iceberg_1_9: name: Iceberg Spark SQL Tests (Iceberg 1.9) needs: changes - # Queue-only by default; PRs need the `run-iceberg-tests` label. + # Nightly by default; PRs need the `run-iceberg-tests` label. if: needs.changes.outputs.iceberg_1_9 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: @@ -402,7 +444,7 @@ jobs: iceberg_1_10: name: Iceberg Spark SQL Tests (Iceberg 1.10) needs: changes - # Queue-only by default; PRs need the `run-iceberg-tests` label. + # Nightly by default; PRs need the `run-iceberg-tests` label. if: needs.changes.outputs.iceberg_1_10 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: @@ -416,8 +458,8 @@ jobs: name: Iceberg Spark SQL Tests (Iceberg 1.11) needs: changes # Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg - # 1.11 is our only Spark 4.1 Iceberg coverage, and it was the PR-tier - # Iceberg job for that reason until every suite moved behind the queue. + # 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one + # Iceberg version the queue runs while 1.8/1.9/1.10 run nightly. if: needs.changes.outputs.iceberg_1_11 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: @@ -493,3 +535,67 @@ jobs: run: | echo "::error::One or more upstream jobs did not succeed. See the results above." exit 1 + + # --------------------------------------------------------------------------- + # nightly_report: a red nightly has no pull request to show up on, and a + # scheduled run only emails whoever last touched the workflow file, so + # without this a regression in the nightly tier sits unnoticed on the + # Actions tab. On the scheduled event, and only when `required_checks` is + # not green, it opens an issue labelled `ci-nightly-failure` listing the + # failed jobs, or comments on the one that is already open so consecutive + # red nights accumulate in one place instead of one issue a day. Closing + # the issue is how the failure is acknowledged; the next red night opens a + # new one. + # + # It sits downstream of the aggregator rather than of the nightly jobs + # themselves so the job list here does not have to be kept in step with + # POLICY. + # --------------------------------------------------------------------------- + nightly_report: + name: Nightly failure report + needs: required_checks + if: always() && github.event_name == 'schedule' && needs.required_checks.result != 'success' + permissions: + actions: read + issues: write + runs-on: ubuntu-slim + steps: + - uses: actions/github-script@v9 + with: + script: | + const label = 'ci-nightly-failure'; + const { owner, repo } = context.repo; + const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`; + const shortSha = context.sha.slice(0, 10); + const day = new Date().toISOString().slice(0, 10); + + const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, { + owner, repo, run_id: context.runId, filter: 'latest', per_page: 100, + }); + const failed = jobs + .filter((job) => job.conclusion === 'failure' || job.conclusion === 'cancelled') + .map((job) => `- [${job.name}](${job.html_url}) (${job.conclusion})`); + + const body = [ + `The nightly CI run against \`main\` at ${shortSha} did not pass: ${runUrl}`, + '', + failed.length ? 'Jobs that did not succeed:' : 'No individual job reported a failure; see the run for details.', + ...failed, + '', + 'The nightly tier runs the Spark and Iceberg versions the merge queue does not', + '(see `POLICY` in `dev/ci/compute-changes.py`). Close this issue once the failure', + 'is understood; the next red nightly opens a new one.', + ].join('\n'); + + const { data: open } = await github.rest.issues.listForRepo({ + owner, repo, labels: label, state: 'open', per_page: 1, + }); + if (open.length > 0) { + await github.rest.issues.createComment({ owner, repo, issue_number: open[0].number, body }); + core.info(`Commented on #${open[0].number}`); + } else { + const { data: issue } = await github.rest.issues.create({ + owner, repo, title: `Nightly CI failed on ${day}`, body, labels: [label], + }); + core.info(`Opened #${issue.number}`); + } diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml index 0a4f47c0c3..58422f40ab 100644 --- a/.github/workflows/pr_build_linux.yml +++ b/.github/workflows/pr_build_linux.yml @@ -34,17 +34,25 @@ name: PR Build (Linux) # # Within the full pipeline, `profiles` picks which Spark profiles the # linux-test matrix runs the Comet test suites against: `pr` is the default -# build profile alone, `all` is every profile, and `queue-only` is the rest. +# build profile alone, `all` is every profile, and `nightly` is the rest. # The rows live in dev/ci/linux-test-profiles.py, which the `lint` job runs so -# that the matrix can be built from its output. The queue runs `all`; a pull -# request runs `pr` unless it carries `run-all-spark-profiles`. See the -# `build_linux_all_profiles` comment in dev/ci/compute-changes.py. +# that the matrix can be built from its output. A pull request and the merge +# queue run `pr`, the nightly run `nightly`, and a pull request carrying +# `run-all-spark-profiles` runs `all`. See the `build_linux_all_profiles` +# comment in dev/ci/compute-changes.py. # -# Keeping both modes in this file is deliberate. The cache keys are the whole -# point of the push tier, and a separate cache-warming workflow would have to -# repeat every one of them; here they cannot drift apart. dev/ci/check-ci-config.py -# pins which jobs survive `cache-refresh-only`, so a job added without the guard -# cannot quietly put the full pipeline back on the push tier. +# `nightly` is also a third mode. Whenever ci.yml passes it, the default +# profile's pipeline has already run at this commit (in the queue and the push +# run for the scheduled event, in the PR tier for a label run), so only `lint`, +# `build-native` and the `linux-test` matrix run; every other job carries +# `if: ${{ inputs.profiles != 'nightly' }}`. +# +# Keeping all three modes in this file is deliberate. The cache keys are the +# whole point of the push tier, and a separate cache-warming workflow would +# have to repeat every one of them; here they cannot drift apart. +# dev/ci/check-ci-config.py pins which jobs survive `cache-refresh-only` and +# which run under `profiles: nightly`, so a job added without either guard +# cannot quietly put the full pipeline back on those tiers. on: workflow_call: inputs: @@ -56,11 +64,11 @@ on: default: false profiles: # Which Spark profiles the linux-test matrix runs. `pr` is the default - # build profile alone, `all` is every profile, `queue-only` is every - # profile except the `pr` one (a `labeled` run, where the PR-tier row - # already ran at this commit). Rows and tiers are in - # dev/ci/linux-test-profiles.py. - description: 'Which Spark profiles linux-test runs: all, pr (the default build profile) or queue-only (the rest)' + # build profile alone, `all` is every profile, `nightly` is every + # profile except the `pr` one (the nightly run, and a `labeled` run, + # where the PR-tier row already ran at this commit). Rows and tiers + # are in dev/ci/linux-test-profiles.py. + description: 'Which Spark profiles linux-test runs: all, pr (the default build profile) or nightly (the rest)' type: string default: 'all' @@ -107,7 +115,7 @@ jobs: # set, including the semantic rules, still runs in lint-java. scalafix-syntactic: name: Lint Scala (syntactic) - if: ${{ !inputs.cache-refresh-only }} + if: ${{ !inputs.cache-refresh-only && inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v7 @@ -148,7 +156,7 @@ jobs: lint-java: needs: lint name: Lint Java (${{ matrix.profile.name }}) - if: ${{ !inputs.cache-refresh-only }} + if: ${{ !inputs.cache-refresh-only && inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -228,7 +236,7 @@ jobs: build-spark-4-1: needs: lint name: Build Spark 4.1, JDK 17 - if: ${{ !inputs.cache-refresh-only }} + if: ${{ !inputs.cache-refresh-only && inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -260,7 +268,7 @@ jobs: celeborn-reflection-compatibility: needs: lint name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility - if: ${{ !inputs.cache-refresh-only }} + if: ${{ !inputs.cache-refresh-only && inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -350,7 +358,9 @@ jobs: - name: Save Cargo cache uses: actions/cache/save@v6 - if: github.ref == 'refs/heads/main' + # The push run is the cache warmer (see the header); a scheduled run + # at the same sha would only re-archive an entry that already exists. + if: github.event_name == 'push' with: path: | ~/.cargo/registry @@ -367,6 +377,7 @@ jobs: linux-test-rust: needs: lint name: ubuntu-latest/rust-test + if: ${{ inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -396,7 +407,9 @@ jobs: - name: Save Cargo cache uses: actions/cache/save@v6 - if: github.ref == 'refs/heads/main' + # The push run is the cache warmer (see the header); a scheduled run + # at the same sha would only re-archive an entry that already exists. + if: github.event_name == 'push' with: path: | ~/.cargo/registry @@ -413,9 +426,9 @@ jobs: matrix: # The Spark profiles live in dev/ci/linux-test-profiles.py and are # selected by the `profiles` input: the default build profile on a - # pull request, all five in the merge queue. The goal of the full list - # is coverage of every Java, Scala and Spark version without testing - # every combination. + # pull request and in the merge queue, the other four in the nightly + # run. The goal of the full list is coverage of every Java, Scala and + # Spark version without testing every combination. profile: ${{ fromJSON(needs.lint.outputs.profile-matrix) }} # Suites are grouped by functional area into balanced buckets so that no test # job runs much longer than ~23 min. See @@ -611,6 +624,7 @@ jobs: verify-benchmark-results-tpch: needs: build-native name: Verify TPC-H Results + if: ${{ inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -675,6 +689,7 @@ jobs: verify-benchmark-results-tpcds: needs: build-native name: Verify TPC-DS Results + if: ${{ inputs.profiles != 'nightly' }} runs-on: ubuntu-24.04 container: image: amd64/rust diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 3b140d27bd..b271bc60f6 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -79,8 +79,9 @@ # Jobs that legitimately stay out of the aggregator's `needs:`. `docs` deploys # to asf-site on push to main; it gates nothing and is never part of a merge # decision, so folding it in would only turn a failed site deploy into a red -# `Required Checks` on main. -AGGREGATOR_EXEMPT = {AGGREGATOR_JOB, "docs"} +# `Required Checks` on main. `nightly_report` runs *after* the aggregator, on +# the scheduled event only, to open an issue when the nightly tier fails. +AGGREGATOR_EXEMPT = {AGGREGATOR_JOB, "docs", "nightly_report"} # Changed-file list -> the set of outputs compute-changes.py must report true. # Every other output must be false. Keep one case per shared build input so a @@ -147,34 +148,54 @@ # derived from POLICY, so that a change to the routing has to be stated twice # and cannot be made by accident. # The PR tier is the Linux build and nothing else. Every Spark SQL and Iceberg -# suite waits for the queue, or for its label. +# suite waits for the queue or the nightly run, or for its label. PR_TIER = {"build_linux", "build_linux_full"} -SPARK_OPT_IN = {"spark_3_5", "spark_4_0", "spark_4_1", "spark_4_1_hive"} -# Spark 3.4 is deprecated and sits outside the queue tier entirely: a label on -# a pull request, or a workflow_dispatch, and nothing else. Keeping it in its -# own set is what makes the `merge_group` case below assert its absence rather -# than quietly accept it coming back. -SPARK_DEPRECATED = {"spark_3_4"} -ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"} -# `build_linux_all_profiles` is the linux-test matrix's non-default Spark -# profiles: part of the Linux build's call, not a job of its own. -BUILD_OPT_IN = { +# The queue adds one Spark version (4.1, the default profile) and one Iceberg +# version (1.11, the only Spark 4.1 coverage), plus the build-level gates. +QUEUE_TIER = PR_TIER | { + "spark_4_1", + "spark_4_1_hive", + "iceberg_1_11", "build_macos", "benchmark", - "build_linux_all_profiles", "delta_gate", "pyarrow_udf", } -QUEUE_TIER = PR_TIER | SPARK_OPT_IN | ICEBERG_OPT_IN | BUILD_OPT_IN -ALL_JOBS = QUEUE_TIER | SPARK_DEPRECATED | {"docs"} +# Every other Spark and Iceberg version, and the linux-test matrix's other +# Spark profiles (`build_linux_all_profiles`, part of the Linux build's call +# rather than a job of its own), run once a night against main instead. +NIGHTLY_TIER = { + "spark_3_5", + "spark_4_0", + "iceberg_1_8", + "iceberg_1_9", + "iceberg_1_10", + "build_linux_all_profiles", +} +# Spark 3.4 is deprecated and sits outside the queue and nightly tiers +# entirely: a label on a pull request, or a workflow_dispatch, and nothing +# else. Keeping it in its own set is what makes the `merge_group` and +# `schedule` cases below assert its absence rather than quietly accept it +# coming back. +SPARK_DEPRECATED = {"spark_3_4"} +# One label opts a pull request into every Iceberg version, whichever tier +# each sits in. +ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"} +ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | {"docs"} +assert not QUEUE_TIER & NIGHTLY_TIER, "a job is queue or nightly, never both" POLICY_CASES = [ # A manual run may exercise anything. ({"name": "workflow_dispatch"}, ALL_JOBS), # The merge queue is the authoritative gate: everything except the site - # deploy, which can only run once the commit is actually on main, and the - # deprecated Spark 3.4 suite, which no longer gates a merge. + # deploy, which can only run once the commit is actually on main, the + # nightly tier, and the deprecated Spark 3.4 suite, which no longer gates + # a merge. ({"name": "merge_group"}, QUEUE_TIER), + # The scheduled run is the nightly tier and nothing else. The queue already + # ran everything in QUEUE_TIER against the tree that is now main, so a + # queue job showing up here is a suite being paid for twice a day. + ({"name": "schedule"}, NIGHTLY_TIER), # Push to main is the site deploy plus the Linux build, which is there to # refresh main's actions/cache entries (see POLICY). `build_linux_full` # must stay out: it is what turns the lints and the test matrix back on, @@ -185,13 +206,13 @@ # opt-in suites stay off without their label. ({"name": "pull_request", "action": "opened", "labels": []}, PR_TIER), ({"name": "pull_request", "action": "synchronize", "labels": []}, PR_TIER), - # Spark 3.5 moved behind the queue; its label is the escape hatch. + # Spark 3.5 runs nightly; its label is the escape hatch. ( {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-3.5-tests"]}, PR_TIER | {"spark_3_5"}, ), - # So did the macOS build and the benchmark compile check, each with its - # own label. Neither label pulls in the other. + # The macOS build and the benchmark compile check are queue-only, each + # with its own label. Neither label pulls in the other. ( {"name": "pull_request", "action": "synchronize", "labels": ["run-macos-tests"]}, PR_TIER | {"build_macos"}, @@ -229,10 +250,10 @@ }, {"build_macos"}, ), - # The linux-test matrix's non-default Spark profiles are queue-only with + # The linux-test matrix's non-default Spark profiles run nightly, with # their own label. On a pushed commit the label adds them to the PR tier's # Linux build call (`profiles: all`); on the `labeled` event alone it is - # the only output set, and ci.yml turns that into `profiles: queue-only` + # the only output set, and ci.yml turns that into `profiles: nightly` # so the default profile, which already ran at this commit, is not repeated. ( {"name": "pull_request", "action": "synchronize", "labels": ["run-all-spark-profiles"]}, @@ -380,6 +401,39 @@ # indented further, and those are expected rather than a reason to exempt the # whole job. CACHE_REFRESH_GUARD = re.compile(r"^ if:.*!\s*inputs\.cache-refresh-only") +# The same file's third mode: with `profiles: nightly` only the jobs the +# linux-test matrix needs run. See check_nightly_scope. +NIGHTLY_JOBS = { + "lint": "publishes the profile matrix that linux-test reads", + "build-native": "builds the native library the matrix loads", + "linux-test": "the matrix itself", +} +NIGHTLY_GUARD = re.compile(r"^ if:.*inputs\.profiles\s*!=\s*'nightly'") +# The `schedule` case in ci.yml's `Detect changes` script, and what has to be +# in it. See check_nightly_base_fallback. +CI_WORKFLOW = WORKFLOWS / "ci.yml" +SCHEDULE_BRANCH = re.compile(r'^(\s+)elif \[\[ "\$EVENT_NAME" == "schedule" \]\]; then\s*$') +# Any base derived from a clock rather than from the previous run. +DATE_BASED_BASE = re.compile(r"--(before|since|after|until)\b") + + +def guarded_jobs(path, guard): + """Return (job ids in `path`, the subset whose job-level `if:` matches `guard`).""" + jobs, guarded, job, in_jobs = [], set(), None, False + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("jobs:"): + in_jobs = True + continue + if not in_jobs or line.lstrip().startswith("#"): + continue + match = JOB_KEY.match(line) + if match: + job = match.group(1) + jobs.append(job) + continue + if job and guard.match(line): + guarded.add(job) + return jobs, guarded CACHE_REFRESH_INPUT = re.compile(r"^\s+cache-refresh-only:\s*\$\{\{") # `profiles:` is passed as a folded scalar (`>-`) whose expression sits on the # next line, so match the key alone. @@ -422,12 +476,12 @@ def check_spark_sql_modules(): def check_linux_test_profiles(): - """`--profiles pr` and `--profiles queue-only` must partition `--profiles all`. + """`--profiles pr` and `--profiles nightly` must partition `--profiles all`. ci.yml maps `build_linux_full` and `build_linux_all_profiles` onto these three values. A profile in neither tier would never run anywhere; one in - both would run twice in the queue. The `pr` tier also has to be the - default build profile and nothing else, which is the whole reason the + both would run twice on a labelled pull request. The `pr` tier also has + to be the default build profile and nothing else, which is the reason the split exists. And the caller has to pass the input at all: its default is `all`, so a dropped `with:` line quietly puts every profile back on the pull request tier. @@ -438,14 +492,14 @@ def check_linux_test_profiles(): failures = [] names = lambda rows: [row["name"] for row in rows] everything = names(module.select("all")) - pr, queue_only = names(module.select("pr")), names(module.select("queue-only")) + pr, nightly = names(module.select("pr")), names(module.select("nightly")) if pr != ["Spark 4.1, JDK 17"]: failures.append(f"the pr tier must be the default build profile alone, got {pr}") - if not queue_only: - failures.append("the queue-only tier is empty (see PROFILES in dev/ci/linux-test-profiles.py)") - if sorted(pr + queue_only) != sorted(everything): + if not nightly: + failures.append("the nightly tier is empty (see PROFILES in dev/ci/linux-test-profiles.py)") + if sorted(pr + nightly) != sorted(everything): failures.append( - f"pr {pr} + queue-only {queue_only} does not partition all {everything} " + f"pr {pr} + nightly {nightly} does not partition all {everything} " f"(see PROFILES in dev/ci/linux-test-profiles.py)" ) if len(set(everything)) != len(everything): @@ -514,6 +568,13 @@ def check_event_policy(): f"mutually exclusive; drop 'pr' if the job is opt-in, or drop " f"the label if it should run on every pull request" ) + # The nightly exists so the queue does not pay for these suites; a job + # in both tiers would run twice a day for one verdict. + if "queue" in tiers and "nightly" in tiers: + failures.append( + f"{job}: POLICY lists both 'queue' and 'nightly'. The queue " + f"already tested the tree the nightly runs against, so pick one" + ) for failure in failures: print(f"event policy: {failure}") return not failures @@ -793,20 +854,7 @@ def check_cache_refresh_scope(): minutes a push, which is what this check exists to notice. """ failures = [] - jobs, guarded, job, in_jobs = [], set(), None, False - for line in CACHE_REFRESH_WORKFLOW.read_text(encoding="utf-8").splitlines(): - if line.startswith("jobs:"): - in_jobs = True - continue - if not in_jobs or line.lstrip().startswith("#"): - continue - match = JOB_KEY.match(line) - if match: - job = match.group(1) - jobs.append(job) - continue - if job and CACHE_REFRESH_GUARD.match(line): - guarded.add(job) + jobs, guarded = guarded_jobs(CACHE_REFRESH_WORKFLOW, CACHE_REFRESH_GUARD) for stale in sorted(set(CACHE_REFRESH_JOBS) - set(jobs)): failures.append( @@ -846,6 +894,117 @@ def check_cache_refresh_scope(): return not failures +def check_nightly_scope(): + """With `profiles: nightly`, pr_build_linux.yml runs the test matrix alone. + + That input value means the default-profile pipeline already ran at this + commit: in the queue and the push run for the nightly, in the PR tier for + a `run-all-spark-profiles` label run. The lints, the Rust tests, the + Spark build and the TPC-H/TPC-DS runs would repeat a verdict, so every + job other than the three the matrix needs carries + `if: ${{ inputs.profiles != 'nightly' }}`. Silent when broken, like the + cache-refresh guard: the nightly just costs 100-odd runner-minutes more. + """ + failures = [] + jobs, guarded = guarded_jobs(CACHE_REFRESH_WORKFLOW, NIGHTLY_GUARD) + for stale in sorted(set(NIGHTLY_JOBS) - set(jobs)): + failures.append( + f"NIGHTLY_JOBS names `{stale}`, which no longer exists in " + f"{CACHE_REFRESH_WORKFLOW}; drop it here, or restore the job" + ) + for name in jobs: + if name in NIGHTLY_JOBS and name in guarded: + failures.append( + f"{CACHE_REFRESH_WORKFLOW}: job `{name}` is listed in " + f"NIGHTLY_JOBS ({NIGHTLY_JOBS[name]}) but carries the " + f"`profiles != 'nightly'` guard, so the nightly test matrix " + f"cannot run" + ) + if name not in NIGHTLY_JOBS and name not in guarded: + failures.append( + f"{CACHE_REFRESH_WORKFLOW}: job `{name}` has no " + f"`if: ${{{{ inputs.profiles != 'nightly' }}}}`, so it repeats " + f"on every nightly a verdict the queue already produced. Add " + f"the guard, or add the job to NIGHTLY_JOBS with the reason " + f"the test matrix needs it" + ) + for failure in failures: + print(f"nightly scope: {failure}") + return not failures + + +def schedule_branch(): + """The body of the `schedule` case in ci.yml's `Detect changes` script. + + Ends at the next branch keyword indented the same as the `elif` itself, so + the `if`/`else` nested inside the case stays part of the body. + """ + body, indent = [], None + for line in CI_WORKFLOW.read_text(encoding="utf-8").splitlines(): + if indent is None: + match = SCHEDULE_BRANCH.match(line) + if match: + indent = match.group(1) + continue + if re.match(rf"^{indent}(elif|else|fi)\b", line): + break + body.append(line) + return body + + +def check_nightly_base_fallback(): + """The nightly must run everything when it has no base to diff against. + + `dev/ci/nightly-base.py` returns the commit the last successful scheduled + run tested, and the nightly diffs `main` against it. When it returns + nothing -- the first nightly, an Actions API error, or a base that has + left `main` -- the only safe base is no base at all: list the whole tree + and let POLICY narrow it to the nightly tier. + + Guessing a narrower base is silently destructive, which is why it is + pinned here. Say the last green nightly was three nights ago, a source + change landed two nights ago and no nightly has covered it, and only docs + have landed since. A one-day window starts after that source change, so + its suites are skipped, the run goes green, and `nightly-base.py` then + hands that green head out as tomorrow's base. The coverage is gone, the + run that dropped it was green, and nothing says so. + """ + failures = [] + body = schedule_branch() + if not body: + failures.append( + f"{CI_WORKFLOW}: no `schedule` case in the `Detect changes` script " + f"(SCHEDULE_BRANCH no longer matches); the nightly is unrouted" + ) + code = [line for line in body if not line.lstrip().startswith("#")] + text = "\n".join(code) + if "dev/ci/nightly-base.py" not in text: + failures.append( + f"{CI_WORKFLOW}: the `schedule` case does not call " + f"dev/ci/nightly-base.py, so the nightly is not based on the last " + f"successful scheduled run and commits can be covered twice or not " + f"at all" + ) + if "git ls-tree -r --name-only HEAD" not in text: + failures.append( + f"{CI_WORKFLOW}: the `schedule` case has no " + f"`git ls-tree -r --name-only HEAD` fallback, so a nightly with no " + f"base does not run the whole tier" + ) + for line in code: + if DATE_BASED_BASE.search(line): + failures.append( + f"{CI_WORKFLOW}: the `schedule` case derives a base from the " + f"clock (`{line.strip()}`). A window that starts after a commit " + f"no nightly has covered yet skips that commit's suites, goes " + f"green, and becomes tomorrow's base. Fall back to the whole " + f"tree instead" + ) + for failure in failures: + print(f"nightly base: {failure}") + return not failures + + if __name__ == "__main__": ok = check_change_filters() ok = check_event_policy() and ok @@ -855,6 +1014,8 @@ def check_cache_refresh_scope(): ok = check_local_actions_have_checkout() and ok ok = check_required_checks() and ok ok = check_cache_refresh_scope() and ok + ok = check_nightly_scope() and ok + ok = check_nightly_base_fallback() and ok if not ok: sys.exit(1) print("CI config checks passed") diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index 5e21e2c68f..799b791624 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -396,6 +396,7 @@ # # "pr" every pull request # "queue" the merge queue, i.e. a merge_group event +# "nightly" the scheduled run against main, once a day # "push" push to main # "label:" a pull request carrying that label # @@ -404,11 +405,21 @@ # requests or opt-in, never both -- and check-ci-config.py rejects a job that # lists both rather than letting the label quietly win. # -# Almost everything is "queue": the merge queue is the authoritative gate, and -# it tests the merge result rather than the PR head. "push" is reserved for -# work that can only happen once a commit is on main. Adding "push" back to a -# test job would make every merge run it twice, once in the queue and once -# after, which is the thing the queue was adopted to avoid. +# The merge queue is the authoritative gate: it tests the merge result rather +# than the PR head, and every "queue" job has to pass before a change lands. +# "nightly" is for the suites that catch a regression on a Spark or Iceberg +# version other than the default one: about 870 of the 1,900 runner-minutes a +# queue run cost in September 2026, and the most common reason a queue run +# went red on a good tree (issue #5870). A regression there is real but rare, +# and a day's delay in seeing it costs less than running the suites on every +# merge. The scheduled run diffs main against the commit the last successful +# scheduled run tested and routes through FILTERS like any other event. A job +# is "queue" or "nightly", never both; check-ci-config.py enforces that. +# +# "push" is reserved for work that can only happen once a commit is on main. +# Adding "push" back to a test job would make every merge run it twice, once +# in the queue and once after, which is the thing the queue was adopted to +# avoid. POLICY = { # The one test job that also runs on push to main, and only because of # actions/cache scoping: a pull request can restore caches saved on its @@ -433,12 +444,13 @@ # The linux-test matrix's Spark profiles other than the default one. The # five profiles cost about the same each, roughly 2,300 runner-minutes a # day apiece on pull requests in mid-September 2026, and together they - # were three quarters of the Linux build. A pull request runs the Comet - # test suites against Spark 4.1 only; the queue runs all five. The - # lint-java matrix still compiles Spark 3.4/3.5/4.0 on every pull request, - # so what waits for the queue is runtime behaviour, not a shim that fails - # to build. ci.yml turns this output into the workflow's `profiles` input. - "build_linux_all_profiles": ["queue", "label:run-all-spark-profiles"], + # were three quarters of the Linux build. A pull request and the queue run + # the Comet test suites against Spark 4.1 only; the nightly run covers the + # other four. The lint-java matrix still compiles Spark 3.4/3.5/4.0 on + # every pull request, so what waits for the nightly is runtime behaviour, + # not a shim that fails to build. ci.yml turns this output into the + # workflow's `profiles` input. + "build_linux_all_profiles": ["nightly", "label:run-all-spark-profiles"], # macOS runners are the scarcest capacity we have, and the Linux build # already covers rustfmt and the Rust/JVM compile on every PR. The label # is for a change that touches platform-specific code. @@ -461,8 +473,11 @@ # demand -- the label on a pull request, or a workflow_dispatch -- so # anyone who wants to check a change against 3.4 still can. "spark_3_4": ["label:run-spark-3.4-tests"], - "spark_3_5": ["queue", "label:run-spark-3.5-tests"], - "spark_4_0": ["queue", "label:run-spark-4.0-tests"], + # Spark 4.1 is the default build profile and the one Spark SQL suite the + # queue runs; 3.5 and 4.0 run nightly, or on a pull request with their + # label. + "spark_3_5": ["nightly", "label:run-spark-3.5-tests"], + "spark_4_0": ["nightly", "label:run-spark-4.0-tests"], # No Spark SQL suite runs on a plain pull request. Spark 4.1 was the last # one in the PR tier, first whole (issue #5870 pulled the sql_hive shards # out) and then catalyst and sql_core alone. What changed is how often a @@ -478,12 +493,12 @@ "label:run-spark-4.1-tests", "label:run-spark-4.1-hive-tests", ], - # Same for Iceberg: 1.11 was the PR-tier version because it is the only - # Spark 4.1 coverage, and it now waits for the queue with the other three. - # One label opts a pull request into all four. - "iceberg_1_8": ["queue", "label:run-iceberg-tests"], - "iceberg_1_9": ["queue", "label:run-iceberg-tests"], - "iceberg_1_10": ["queue", "label:run-iceberg-tests"], + # Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is + # the one Iceberg version the queue runs; the three older versions run + # nightly. One label opts a pull request into all four. + "iceberg_1_8": ["nightly", "label:run-iceberg-tests"], + "iceberg_1_9": ["nightly", "label:run-iceberg-tests"], + "iceberg_1_10": ["nightly", "label:run-iceberg-tests"], "iceberg_1_11": ["queue", "label:run-iceberg-tests"], } @@ -508,6 +523,8 @@ def event_allows(job, event): return "push" in tiers if name == "merge_group": return "queue" in tiers + if name == "schedule": + return "nightly" in tiers if name != "pull_request": return False diff --git a/dev/ci/linux-test-profiles.py b/dev/ci/linux-test-profiles.py index c2162aaf45..741ef7d3b1 100644 --- a/dev/ci/linux-test-profiles.py +++ b/dev/ci/linux-test-profiles.py @@ -20,12 +20,12 @@ # # The five Spark profiles cost about the same each, and together they are # three quarters of what the Linux build spends on a pull request. Only one of -# them is the default build profile, so a pull request runs the Comet test -# suites against that one and the merge queue runs all five. A job-level `if:` -# cannot see `matrix`, so the selection has to happen before the matrix is -# expanded: the `lint` job runs this script and publishes the result as a job -# output that `linux-test` reads with `fromJSON`, the same way -# spark-sql-modules.py picks the Spark SQL shards. +# them is the default build profile, so a pull request and the merge queue run +# the Comet test suites against that one and the nightly run covers the other +# four. A job-level `if:` cannot see `matrix`, so the selection has to happen +# before the matrix is expanded: the `lint` job runs this script and publishes +# the result as a job output that `linux-test` reads with `fromJSON`, the same +# way spark-sql-modules.py picks the Spark SQL shards. # # `lint-java` keeps its own literal profile list. It compiles every profile it # can on every pull request (about five minutes each), which is what keeps a @@ -33,7 +33,7 @@ # runtime suites move behind it. # # Usage: -# linux-test-profiles.py --profiles all|pr|queue-only --github-output $GITHUB_OUTPUT +# linux-test-profiles.py --profiles all|pr|nightly --github-output $GITHUB_OUTPUT # linux-test-profiles.py --profiles pr (prints the matrix JSON) import argparse @@ -42,31 +42,26 @@ from pathlib import Path # `tier` is what --profiles selects on: "pr" rows run on every pull request -# and in the queue, "queue" rows only in the queue (or with the +# and in the queue, "nightly" rows only in the nightly run (or with the # `run-all-spark-profiles` label). The goal of the list is coverage of every # Java, Scala and Spark version without testing every combination. PROFILES = [ - {"name": "Spark 3.4, JDK 17, Scala 2.12", "java_version": "17", "maven_opts": "-Pspark-3.4 -Pscala-2.12", "tier": "queue"}, - {"name": "Spark 3.5, JDK 17, Scala 2.13", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.13", "tier": "queue"}, - {"name": "Spark 4.0, JDK 21", "java_version": "21", "maven_opts": "-Pspark-4.0", "tier": "queue"}, + {"name": "Spark 3.4, JDK 17, Scala 2.12", "java_version": "17", "maven_opts": "-Pspark-3.4 -Pscala-2.12", "tier": "nightly"}, + {"name": "Spark 3.5, JDK 17, Scala 2.13", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.13", "tier": "nightly"}, + {"name": "Spark 4.0, JDK 21", "java_version": "21", "maven_opts": "-Pspark-4.0", "tier": "nightly"}, # The default build profile, and the one a contributor builds locally. {"name": "Spark 4.1, JDK 17", "java_version": "17", "maven_opts": "-Pspark-4.1", "tier": "pr"}, - {"name": "Spark 4.2, JDK 17", "java_version": "17", "maven_opts": "-Pspark-4.2", "tier": "queue"}, + {"name": "Spark 4.2, JDK 17", "java_version": "17", "maven_opts": "-Pspark-4.2", "tier": "nightly"}, ] -SELECTORS = ("all", "pr", "queue-only") +SELECTORS = ("all", "pr", "nightly") def select(profiles): """Return the matrix rows for a --profiles value, or raise ValueError.""" if profiles not in SELECTORS: raise ValueError(f"--profiles must be one of {', '.join(SELECTORS)}, got {profiles!r}") - if profiles == "all": - rows = PROFILES - elif profiles == "pr": - rows = [row for row in PROFILES if row["tier"] == "pr"] - else: - rows = [row for row in PROFILES if row["tier"] == "queue"] + rows = [row for row in PROFILES if profiles == "all" or row["tier"] == profiles] return [{key: value for key, value in row.items() if key != "tier"} for row in rows] @@ -75,7 +70,7 @@ def main(argv): parser.add_argument( "--profiles", default="all", - help="all, pr (the profiles every pull request runs) or queue-only (the ones it does not)", + help="all, pr (the profiles every pull request runs) or nightly (the ones it does not)", ) parser.add_argument("--github-output", type=Path, help="append matrix= to this $GITHUB_OUTPUT file") args = parser.parse_args(argv) diff --git a/dev/ci/nightly-base.py b/dev/ci/nightly-base.py new file mode 100644 index 0000000000..0752999f6c --- /dev/null +++ b/dev/ci/nightly-base.py @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Print the head sha of the most recent successful scheduled run of ci.yml, +# or nothing. +# +# The nightly diffs main against that commit, so every commit that landed +# since the last green nightly is covered exactly once, and a red nightly +# keeps the regressing commits in scope until a green one supersedes it. When +# this prints nothing -- the first scheduled run, or an API error, which is +# reported as a warning rather than failing the run -- the `changes` job in +# ci.yml has no base it can trust and runs the whole nightly tier instead. +# There is deliberately no narrower fallback: a guessed base that starts after +# a commit no nightly has covered yet would skip that commit's suites, let the +# run go green, and then become tomorrow's base, dropping the coverage for +# good. +# +# Needs GITHUB_TOKEN with `actions: read`, plus the GITHUB_REPOSITORY and +# GITHUB_API_URL variables every job has. + +import json +import os +import sys +import urllib.error +import urllib.request + + +def main(): + repo = os.environ["GITHUB_REPOSITORY"] + api = os.environ.get("GITHUB_API_URL", "https://api.github.com") + url = f"{api}/repos/{repo}/actions/workflows/ci.yml/runs?event=schedule&status=success&per_page=1" + request = urllib.request.Request( + url, + headers={ + "Accept": "application/vnd.github+json", + "Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}", + "X-GitHub-Api-Version": "2022-11-28", + }, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + runs = json.load(response)["workflow_runs"] + except (urllib.error.URLError, OSError, KeyError, ValueError) as e: + # stderr, so the caller's command substitution stays empty. + print( + f"::warning::could not list previous scheduled runs ({e}); " + "running the whole nightly tier", + file=sys.stderr, + ) + return 0 + if runs: + print(runs[0]["head_sha"]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 28d1ccc992..20f2eedf80 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -25,32 +25,35 @@ how to opt a pull request into a suite the PR tier skips, and what to do when a mechanics behind the configuration are documented in [.github/workflows/README.md](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/README.md). -## Two tiers +## Three tiers A single umbrella workflow, `.github/workflows/ci.yml`, orchestrates everything. It runs cheap preflight checks first (license headers, Markdown formatting, workflow linting, the CI config checks), computes which heavy jobs the changed files are relevant to, and fans out to those jobs. Which jobs run also depends on the event: -| Suite | Pull request | Merge queue | -| ------------------------------------------------- | ------------ | ----------- | -| Linux build, lint, Rust tests, TPC-H/TPC-DS | yes | yes | -| Comet test suites, Spark 4.1 | yes | yes | -| Comet test suites, Spark 3.4 / 3.5 / 4.0 / 4.2 | with label | yes | -| Spark SQL tests, Spark 4.1, catalyst and sql_core | with label | yes | -| Spark SQL tests, Spark 4.1, sql_hive | with label | yes | -| Iceberg Spark SQL tests, Iceberg 1.11 | with label | yes | -| macOS build and Comet test suites | with label | yes | -| Benchmark compile and lint check | with label | yes | -| Delta contrib build gate | with label | yes | -| PyArrow UDF tests, Spark 4.0 / 4.1 / 4.2 | with label | yes | -| Spark SQL tests, Spark 3.5 / 4.0 | with label | yes | -| Iceberg Spark SQL tests, Iceberg 1.8 / 1.9 / 1.10 | with label | yes | -| Spark SQL tests, Spark 3.4 | with label | no | +| Suite | Pull request | Merge queue | Nightly | +| ------------------------------------------------- | ------------ | ----------- | ------- | +| Linux build, lint, Rust tests, TPC-H/TPC-DS | yes | yes | no | +| Comet test suites, Spark 4.1 | yes | yes | no | +| Spark SQL tests, Spark 4.1, catalyst and sql_core | with label | yes | no | +| Spark SQL tests, Spark 4.1, sql_hive | with label | yes | no | +| Iceberg Spark SQL tests, Iceberg 1.11 | with label | yes | no | +| macOS build and Comet test suites | with label | yes | no | +| Benchmark compile and lint check | with label | yes | no | +| Delta contrib build gate | with label | yes | no | +| PyArrow UDF tests, Spark 4.0 / 4.1 / 4.2 | with label | yes | no | +| Comet test suites, Spark 3.4 / 3.5 / 4.0 / 4.2 | with label | no | yes | +| Spark SQL tests, Spark 3.5 / 4.0 | with label | no | yes | +| Iceberg Spark SQL tests, Iceberg 1.8 / 1.9 / 1.10 | with label | no | yes | +| Spark SQL tests, Spark 3.4 | with label | no | no | The **PR tier** is the fast feedback loop while a change is being iterated on. The **queue -tier** is the authoritative gate: everything the PR tier runs plus the remaining suites, evaluated -against the merge result rather than the pull request head. Nothing in the queue tier runs again +tier** is the authoritative gate: everything the PR tier runs plus the suites that must pass before +a change lands, evaluated against the merge result rather than the pull request head. The queue +runs one Spark version and one Iceberg version, the ones the default build profile targets. The +**nightly tier** runs the other Spark and Iceberg versions once a day against `main` as it stands; +see [Nightly runs](#nightly-runs) below. Nothing in the queue tier runs again on push to `main`, because the queue already tested the exact tree that landed. The one exception is the Linux build, which also runs on push so that the dependency caches on `main` stay fresh: a pull request can only restore caches saved on its own branch or on `main`, and the queue's @@ -100,9 +103,10 @@ Each suite outside the PR tier has a label that runs it on a pull request: | `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 | | `run-iceberg-tests` | Iceberg Spark SQL tests against every Iceberg version | -For every suite except Spark 3.4 the label only brings the run forward; the queue would have run -it anyway before the change landed. For Spark 3.4 the label is the only way the suite runs on a -pull request at all. +For a queue-tier suite the label only brings the run forward; the queue would have run it anyway +before the change landed. For a nightly-tier suite the label is what gets a verdict _before_ the +change lands: without it, the first run is the nightly after the merge. For Spark 3.4 the label is +the only way the suite runs at all. Apply a label from the pull request sidebar, or from the command line: @@ -115,13 +119,14 @@ executes only the suite the label gates; the PR tier already ran at that commit repeated. Its aggregate verdict is published as `Required Checks (label run)` rather than `Required Checks`, so it can be read alongside the commit run without replacing it. -For every suite except Spark 3.4, that separate name costs nothing: the merge queue runs the suite -again before the change lands, so a failure a label run surfaced still blocks the merge later. -Spark 3.4 has no queue run behind it. A red `Required Checks (label run)` leaves an earlier green -`Required Checks` in place and the pull request mergeable, so treat a label run as feedback to read, -not as a gate. For Spark 3.4 to count toward the required verdict the label has to already be on -the pull request when a commit is pushed — which is what the next push gives you, since the label -stays applied. +For a queue-tier suite, that separate name costs nothing: the merge queue runs the suite again +before the change lands, so a failure a label run surfaced still blocks the merge later. A +nightly-tier suite and Spark 3.4 have no queue run behind them. A red `Required Checks (label run)` +leaves an earlier green `Required Checks` in place and the pull request mergeable, so treat a label +run as feedback to read, not as a gate. For one of those suites to count toward the required +verdict the label has to already be on the pull request when a commit is pushed, which is what the +next push gives you, since the label stays applied. A nightly-tier regression that does slip +through is caught by the next nightly run and filed as an issue; a Spark 3.4 one is not. The label stays on the pull request, so every later push runs the suite as part of the normal PR run. Remove the label once it has served its purpose. To re-run the suite at the same commit, @@ -210,6 +215,35 @@ through these in order: A pull request evicted from the queue has to be queued again by hand. Merge when ready is not re-armed automatically. +## Nightly runs + +`ci.yml` also runs on a schedule, at 06:00 UTC every day, against what landed on `main` since the +last successful scheduled run. That run executes only the nightly tier: the Comet test suites against the Spark +profiles other than 4.1, the Spark SQL suites for Spark 3.5 and 4.0, and the Iceberg suites for +1.8, 1.9 and 1.10, each only when the day's changes touched files it covers. The queue already ran +everything else against the same tree, so nothing in the queue tier is repeated. A day with no +merges, or with only documentation changes, runs nothing. + +If the previous run cannot be found — the first nightly, an unreachable Actions API, or a base that +is no longer on `main` — the run has nothing to diff against and runs the whole nightly tier +instead of guessing at a range. + +A red nightly has no pull request to show up on, so the run opens an issue labelled +`ci-nightly-failure` that links the run and lists the jobs that failed. If one of those issues is +already open, the run comments on it instead, so a failure that persists across several nights +stays in one place. When you pick up a nightly failure: + +1. **Find the commit.** The issue names the `main` commit the run tested, and the run's + `Detect changes` log names the base it diffed against, which is the commit the previous green + nightly tested. The regression is in that range. +2. **Reproduce it on a pull request.** Open the fix as a pull request and apply the label for + the suite that failed, so the same suite runs against the fix before it lands. +3. **Close the issue** once the failure is understood, whether it was fixed, excluded, or found to + be infrastructure. The next red nightly opens a new issue. + +Dispatching `ci.yml` from the Actions page with **Run workflow** runs every tier, including the +nightly suites, if you need a result before the next scheduled run. + ## Reproducing a suite failure locally The Spark SQL suites outside the PR tier run Spark's own test suite against Comet, with the diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md index f825cde080..b5752e0863 100644 --- a/docs/source/contributor-guide/development.md +++ b/docs/source/contributor-guide/development.md @@ -667,10 +667,10 @@ excluded and how to run it. Run a manual suite with: Only add a suite to that list with a good reason; the default is that a new suite runs in CI. -On a pull request the Linux build runs these suites against the default Spark profile (4.1) only; -the merge queue runs them against every Spark profile, and the macOS suites only run in the merge -queue by default. See [Continuous Integration](ci.md) for the two tiers and the labels that opt a -pull request into a queue-only suite. +On a pull request and in the merge queue the Linux build runs these suites against the default +Spark profile (4.1) only; the nightly run covers the other Spark profiles, and the macOS suites +only run in the merge queue by default. See [Continuous Integration](ci.md) for the three tiers +and the labels that opt a pull request into a queue-only or nightly suite. ### Pre-PR Summary diff --git a/docs/source/contributor-guide/iceberg-spark-tests.md b/docs/source/contributor-guide/iceberg-spark-tests.md index 8e45e43446..09374a041f 100644 --- a/docs/source/contributor-guide/iceberg-spark-tests.md +++ b/docs/source/contributor-guide/iceberg-spark-tests.md @@ -100,12 +100,12 @@ diff must be generated against its own tag. The `iceberg_spark_test_.yml` workflows apply these diffs and run the three Gradle targets above against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against Spark 3.4.3; Iceberg 1.9.1 and 1.10.0 -run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. All four versions run in the -merge queue, or earlier on a pull request labeled `run-iceberg-tests`; none runs on an unlabeled -pull request. All caller +run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. Iceberg 1.11.0 runs in the +merge queue; 1.8.1, 1.9.1 and 1.10.0 run once a night against `main`. All four run earlier on a +pull request labeled `run-iceberg-tests`; none runs on an unlabeled pull request. All caller workflows delegate to `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See [.github/workflows/README.md](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/README.md) -for how the pull-request and merge-queue tiers differ. +for how the pull-request, merge-queue and nightly tiers differ. The core Spark test target runs in four independent workers. The workflow passes `dev/ci/iceberg-test-shards.gradle` as a Gradle init script: one worker runs the long @@ -132,4 +132,5 @@ and failure propagation. The fixture does not compile Spark or Iceberg. Apply the `run-iceberg-tests` label to a pull request whenever it touches the Iceberg scan or write path, reflection code (`org.apache.comet.iceberg.IcebergReflection`), or other logic whose behavior can differ across Iceberg versions. The Comet test suites in the Linux build do not exercise Iceberg's -own Spark tests, so without the label the first Iceberg verdict is the merge queue's. +own Spark tests, so without the label the first Iceberg 1.11 verdict is the merge queue's, and the +first verdict on the older Iceberg versions is the nightly run's, after the change has landed. diff --git a/docs/source/contributor-guide/spark-sql-tests.md b/docs/source/contributor-guide/spark-sql-tests.md index c4834bcff9..acfb8fcedc 100644 --- a/docs/source/contributor-guide/spark-sql-tests.md +++ b/docs/source/contributor-guide/spark-sql-tests.md @@ -169,9 +169,11 @@ git diff v3.5.6 > ../datafusion-comet/dev/diffs/3.5.6.diff ## Running Tests in CI The easiest way to run the tests is to open a pull request against Comet and let CI run them. Spark -3.5, 4.0 and 4.1 all run in the merge queue, or earlier on a pull request carrying the -`run-spark-3.5-tests` / `run-spark-4.0-tests` / `run-spark-4.1-tests` label. No Spark SQL suite -runs on an unlabeled pull request. +4.1 runs in the merge queue; Spark 3.5 and 4.0 run once a night against `main`. Each also runs +earlier on a pull request carrying the `run-spark-3.5-tests` / `run-spark-4.0-tests` / +`run-spark-4.1-tests` label. No Spark SQL suite runs on an unlabeled pull request. Apply the 3.5 or +4.0 label when a change could behave differently on those versions, since without it the first +verdict is the nightly run's, after the change has landed. Spark 3.4 is deprecated and is not run by the merge queue. It runs only when a pull request carries the `run-spark-3.4-tests` label, or when `ci.yml` is dispatched manually from the Actions page.