diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 7d591f8a99..81085bc618 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -41,8 +41,20 @@ and has to run after the commit is on main, and `pr_build_linux`, because of `actions/cache` scoping. A pull request can only restore caches saved on its own branch or on `main`, and the queue runs on a throwaway `gh-readonly-queue/*` branch whose caches are deleted with it. Without a push -run, a `Cargo.lock` or `pom.xml` change would leave the cargo-registry, Maven -and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. +run, a `Cargo.lock` or `pom.xml` change would leave the cargo-ci, cargo-debug, +Maven and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. + +Warming those caches is the only thing the push run is for, so on `push` the +Linux build runs in **cache-refresh-only** mode: `build-native`, +`linux-test-rust` and the two TPC-H/TPC-DS jobs, each stopping once its cache +entry is written, and nothing else. The lints, the 5x4 `linux-test` matrix and +the TPC query runs are skipped, which takes the push tier from 587 +runner-minutes to about 73. Two POLICY outputs express this: `build_linux` +says whether the workflow runs at all, `build_linux_full` whether it runs the +lints and tests too, and `ci.yml` folds the second into the workflow's +`cache-refresh-only` input. `dev/ci/check-ci-config.py` fails if a job is added +to `pr_build_linux.yml` without either the guard or an entry in +`CACHE_REFRESH_JOBS` naming the cache it writes. See issue #5929. ``` pull_request | merge_group | push to main | workflow_dispatch @@ -68,7 +80,7 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. v v v PR + queue tier push to main only queue tier, or PR with label --------------- ----------------- --------------------------- - pr_build_linux (+ push, for cache) docs pr_build_macos run-macos-tests + pr_build_linux (+ push, cache only) docs pr_build_macos run-macos-tests spark_4_1 (catalyst + sql_core) pr_benchmark_check run-benchmark-check iceberg_1_11 spark_4_1 sql_hive run-spark-4.1-hive-tests spark_3_5 run-spark-3.5-tests @@ -99,7 +111,7 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. | -------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | `preflight` | every PR / merge group / push / dispatch / label | none (always runs) | | `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` | -| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` | +| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full` | `dev/ci/compute-changes.py` | | `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` | | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only | | `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | @@ -200,6 +212,13 @@ in `dev/ci/compute-changes.py`: Moving a suite between the PR and queue tiers is a one-word edit to `POLICY`. +An output does not have to map one-to-one onto a job. Two outputs can feed a +single call when part of a workflow belongs in a different tier from the rest: +`spark_4_1` / `spark_4_1_hive` select which module shards the one Spark 4.1 +build runs, and `build_linux` / `build_linux_full` select whether the Linux +build runs everything or only the jobs that populate `main`'s caches. Both +share their `FILTERS` list by assignment so the two entries cannot drift. + So adding a suite, moving sources, or changing when something runs is an edit to one of those two tables, not to ten `${{ }}` expressions. Keeping the policy in Python is also what makes it testable: GitHub expressions cannot be diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c82ee4fbd3..e5c9d712f2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,7 +38,9 @@ # push to main would double the cost of every merge. Two exceptions: `docs` # deploys to asf-site and must run after the commit is on main, and the Linux # build runs on push so that main's actions/cache entries stay fresh (caches -# saved on the queue's throwaway branch are deleted with it). +# saved on the queue's throwaway branch are deleted with it). That second +# exception runs in `cache-refresh-only` mode: the cache writers and nothing +# else, because the queue has already tested the tree that landed. name: CI @@ -157,6 +159,7 @@ jobs: runs-on: ubuntu-slim outputs: build_linux: ${{ steps.compute.outputs.build_linux }} + build_linux_full: ${{ steps.compute.outputs.build_linux_full }} build_macos: ${{ steps.compute.outputs.build_macos }} benchmark: ${{ steps.compute.outputs.benchmark }} docs: ${{ steps.compute.outputs.docs }} @@ -233,6 +236,14 @@ jobs: needs: changes if: needs.changes.outputs.build_linux == 'true' uses: ./.github/workflows/pr_build_linux.yml + # Two POLICY outputs feed one call, the same shape as spark_4_1 below. + # `build_linux` decides whether the workflow runs at all; `build_linux_full` + # decides whether it runs the lints and the 5x4 test matrix as well as the + # jobs that populate main's actions/cache entries. Only push to main sets + # the first without the second, which is the whole point: the queue has + # already tested that tree, so the push run is there for the caches alone. + with: + cache-refresh-only: ${{ needs.changes.outputs.build_linux_full != 'true' }} pr_build_macos: name: PR Build (macOS) diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml index ff864ea751..750a1591ac 100644 --- a/.github/workflows/pr_build_linux.yml +++ b/.github/workflows/pr_build_linux.yml @@ -19,8 +19,33 @@ name: PR Build (Linux) # Reusable: invoked by ci.yml. Triggering, path filters, and concurrency # live in the umbrella workflow. +# +# Two modes, selected by `cache-refresh-only`: +# +# false the full pipeline: the lints, the 5x4 linux-test matrix, the Rust +# tests and the TPC-H/TPC-DS correctness runs. Pull requests and the +# merge queue. +# true only the jobs that write an actions/cache entry, and each only far +# enough to populate it. This is the push-to-main tier, where the +# queue has already tested the exact tree that landed and the sole +# remaining purpose of the run is to leave main's caches warm for the +# next pull request. See the `build_linux` / `build_linux_full` +# comments in dev/ci/compute-changes.py for why that tier exists. +# +# Keeping both modes in this file is deliberate. The cache keys are the whole +# point of the push tier, and a separate cache-warming workflow would have to +# repeat every one of them; here they cannot drift apart. dev/ci/check-ci-config.py +# pins which jobs survive `cache-refresh-only`, so a job added without the guard +# cannot quietly put the full pipeline back on the push tier. on: workflow_call: + inputs: + cache-refresh-only: + description: >- + Populate main's actions/cache entries without running any lint or + test. Set by ci.yml on push to main. + type: boolean + default: false env: RUST_VERSION: stable @@ -32,7 +57,9 @@ env: jobs: - # Fast lint check - gates all other jobs + # Fast lint check - gates all other jobs. Runs in cache-refresh-only mode too: + # it is 40 seconds, and `build-native` and `linux-test-rust` both `needs:` it, + # so skipping it would skip the two jobs the push tier exists to run. lint: name: Lint runs-on: ubuntu-24.04 @@ -54,6 +81,7 @@ jobs: # set, including the semantic rules, still runs in lint-java. scalafix-syntactic: name: Lint Scala (syntactic) + if: ${{ !inputs.cache-refresh-only }} runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v7 @@ -94,6 +122,7 @@ jobs: lint-java: needs: lint name: Lint Java (${{ matrix.profile.name }}) + if: ${{ !inputs.cache-refresh-only }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -173,6 +202,7 @@ jobs: build-spark-4-1: needs: lint name: Build Spark 4.1, JDK 17 + if: ${{ !inputs.cache-refresh-only }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -204,6 +234,7 @@ jobs: celeborn-reflection-compatibility: needs: lint name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility + if: ${{ !inputs.cache-refresh-only }} runs-on: ubuntu-24.04 container: image: amd64/rust @@ -247,7 +278,9 @@ jobs: -Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \ -DfailIfNoTests=false - # Build native library once and share with all test jobs + # Build native library once and share with all test jobs. Also the owner of + # main's `cargo-ci` cache entry, which is why it runs in cache-refresh-only + # mode: rebuilding native/target from cold costs every pull request ~20 min. build-native: needs: lint name: Build Native Library @@ -299,7 +332,12 @@ jobs: native/target key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} - # Run Rust tests (runs in parallel with build-native, uses debug builds) + # Run Rust tests (runs in parallel with build-native, uses debug builds). + # Owns main's `cargo-debug` cache entry, so it runs in cache-refresh-only mode + # as well. The tests themselves are a small slice of its runtime -- almost all + # of it is the compile that populates the cache -- so there is nothing to gain + # from a compile-only variant, and running them keeps the cache contents + # identical to what a pull request restores. linux-test-rust: needs: lint name: ubuntu-latest/rust-test @@ -342,6 +380,7 @@ jobs: linux-test: needs: build-native + if: ${{ !inputs.cache-refresh-only }} strategy: matrix: # the goal with these profiles is to get coverage of all Java, Scala, and Spark @@ -551,7 +590,11 @@ jobs: upload-test-reports: true skip-native-build: true - # TPC-H correctness test - verifies benchmark queries produce correct results + # TPC-H correctness test - verifies benchmark queries produce correct results. + # Runs in cache-refresh-only mode for its two cache entries: the SF=1 dataset, + # which is expensive to regenerate, and main's shared `java-maven` entry. Only + # the query run is skipped there; everything up to and including data + # generation still has to happen for the dataset cache to exist. verify-benchmark-results-tpch: needs: build-native name: Verify TPC-H Results @@ -607,11 +650,15 @@ jobs: cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass="org.apache.spark.sql.GenTPCHData" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="--location `pwd`/.. --scaleFactor 1 --numPartitions 1 --overwrite" - name: Run TPC-H queries + if: ${{ !inputs.cache-refresh-only }} run: | SPARK_HOME=`pwd` SPARK_TPCH_DATA=`pwd`/tpch/sf1_parquet ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCHQuerySuite test # TPC-DS correctness tests - verifies benchmark queries produce correct results. - # The three join strategies run sequentially in one job so the project is built once. + # The three join strategies run sequentially in one job so the project is built + # once. Runs in cache-refresh-only mode for the same reason as the TPC-H job; + # there the three query passes are skipped and only the dataset generation and + # the Maven cache remain. verify-benchmark-results-tpcds: needs: build-native name: Verify TPC-DS Results @@ -681,6 +728,7 @@ jobs: cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass="org.apache.spark.sql.GenTPCDSData" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="--dsdgenDir `pwd`/../tpcds-kit/tools --location `pwd`/../tpcds-sf-1 --scaleFactor 1 --numPartitions 1" - name: Run TPC-DS queries (Sort merge join) + if: ${{ !inputs.cache-refresh-only }} run: | SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test env: @@ -689,6 +737,7 @@ jobs: spark.sql.join.preferSortMergeJoin=true - name: Run TPC-DS queries (Broadcast hash join) + if: ${{ !inputs.cache-refresh-only }} run: | SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test env: @@ -696,6 +745,7 @@ jobs: spark.sql.autoBroadcastJoinThreshold=10485760 - name: Run TPC-DS queries (Shuffled hash join) + if: ${{ !inputs.cache-refresh-only }} run: | SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test env: diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 989dbc820b..cf9ec17866 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -# Guards five CI invariants that are silent when broken: +# Guards six CI invariants that are silent when broken: # # 1. Change-filter routing. dev/ci/compute-changes.py decides which heavy # jobs run. A file that a job depends on but that no filter lists makes @@ -50,6 +50,12 @@ # run only under an input or a label can carry that for a long time # before anyone runs them. # +# 6. Push-tier scope. On push to main, ci.yml calls pr_build_linux.yml with +# `cache-refresh-only`, which reduces it to the jobs that write an +# actions/cache entry; the merge queue already tested that tree. A job +# added to that workflow without the guard starts running on every push +# again and nothing fails, so nothing tells you. +# # Run from the repository root: python3 dev/ci/check-ci-config.py import importlib.util @@ -81,6 +87,7 @@ # filter deletion cannot pass unnoticed. BUILD_JOBS = { "build_linux", + "build_linux_full", "build_macos", "spark_3_4", "spark_3_5", @@ -105,7 +112,7 @@ ([".github/actions/upload-artifact-retry/action.yaml"], BUILD_JOBS), ([".github/actions/download-artifact-retry/action.yaml"], BUILD_JOBS), # The Maven bootstrap composite is called only from pr_build_linux.yml. - ([".github/actions/maven-bootstrap/action.yaml"], {"build_linux"}), + ([".github/actions/maven-bootstrap/action.yaml"], {"build_linux", "build_linux_full"}), # Spot checks that the additions above did not widen unrelated routes. (["docs/source/user-guide/overview.md"], {"docs"}), (["native/core/benches/parquet_read.rs"], {"benchmark"}), @@ -115,7 +122,7 @@ # where "allowed" ignores path filters. Written out longhand rather than # derived from POLICY, so that a change to the routing has to be stated twice # and cannot be made by accident. -PR_TIER = {"build_linux", "spark_4_1", "iceberg_1_11"} +PR_TIER = {"build_linux", "build_linux_full", "spark_4_1", "iceberg_1_11"} SPARK_OPT_IN = {"spark_3_5", "spark_4_0", "spark_4_1_hive"} # Spark 3.4 is deprecated and sits outside the queue tier entirely: a label on # a pull request, or a workflow_dispatch, and nothing else. Keeping it in its @@ -135,8 +142,10 @@ # deprecated Spark 3.4 suite, which no longer gates a merge. ({"name": "merge_group"}, QUEUE_TIER), # Push to main is the site deploy plus the Linux build, which is there to - # refresh main's actions/cache entries (see POLICY). Any other test job - # showing up here means every merge is paying for it twice. + # refresh main's actions/cache entries (see POLICY). `build_linux_full` + # must stay out: it is what turns the lints and the test matrix back on, + # and the queue has already run those against the tree that landed. Any + # other test job showing up here means every merge is paying for it twice. ({"name": "push"}, {"docs", "build_linux"}), # A plain pull request: the PR tier only. docs must never run here, and the # opt-in suites stay off without their label. @@ -257,6 +266,24 @@ LOCAL_ACTION_USES = re.compile(r"uses:\s*(\./\.github/actions/\S+)") CHECKOUT_USES = re.compile(r"uses:\s*actions/checkout@") +# pr_build_linux.yml runs in two modes; see its header. These are the jobs that +# must survive `cache-refresh-only`, because each one writes an actions/cache +# entry that main needs warm for the next pull request. Anything else in that +# file has to carry the guard. +CACHE_REFRESH_WORKFLOW = WORKFLOWS / "pr_build_linux.yml" +CACHE_REFRESH_JOBS = { + "lint": "gates build-native and linux-test-rust, and costs 40 seconds", + "build-native": "writes the cargo-ci cache (native/target, CI profile)", + "linux-test-rust": "writes the cargo-debug cache (native/target, debug)", + "verify-benchmark-results-tpch": "writes the TPC-H SF=1 dataset and java-maven caches", + "verify-benchmark-results-tpcds": "writes the TPC-DS SF=1 dataset and java-maven caches", +} +# Job-level `if:` only: step-level guards inside the two verify jobs are +# indented further, and those are expected rather than a reason to exempt the +# whole job. +CACHE_REFRESH_GUARD = re.compile(r"^ if:.*!\s*inputs\.cache-refresh-only") +CACHE_REFRESH_INPUT = re.compile(r"^\s+cache-refresh-only:\s*\$\{\{") + def load_filters(): spec = importlib.util.spec_from_file_location("compute_changes", "dev/ci/compute-changes.py") @@ -606,6 +633,75 @@ def check_required_checks(): return not failures +def check_cache_refresh_scope(): + """Every job in pr_build_linux.yml is either a cache writer or guarded. + + On push to main the merge queue has already tested the exact tree that + landed, so the only thing left for that run to do is leave main's + actions/cache entries warm -- a pull request can restore caches saved on + its own branch or on main and nowhere else, and the queue's throwaway + branch takes its own with it. ci.yml therefore calls the workflow with + `cache-refresh-only` on push, and every job that is not a cache writer + has to opt out with `if: ${{ !inputs.cache-refresh-only }}`. + + A job added without the guard runs on every push again. Nothing fails when + that happens; the runner bill just quietly goes back up by up to ~500 + minutes a push, which is what this check exists to notice. + """ + failures = [] + jobs, guarded, job, in_jobs = [], set(), None, False + for line in CACHE_REFRESH_WORKFLOW.read_text(encoding="utf-8").splitlines(): + if line.startswith("jobs:"): + in_jobs = True + continue + if not in_jobs or line.lstrip().startswith("#"): + continue + match = JOB_KEY.match(line) + if match: + job = match.group(1) + jobs.append(job) + continue + if job and CACHE_REFRESH_GUARD.match(line): + guarded.add(job) + + for stale in sorted(set(CACHE_REFRESH_JOBS) - set(jobs)): + failures.append( + f"CACHE_REFRESH_JOBS names `{stale}`, which no longer exists in " + f"{CACHE_REFRESH_WORKFLOW}; drop it here, or restore the job" + ) + for name in jobs: + if name in CACHE_REFRESH_JOBS and name in guarded: + failures.append( + f"{CACHE_REFRESH_WORKFLOW}: job `{name}` is listed in " + f"CACHE_REFRESH_JOBS ({CACHE_REFRESH_JOBS[name]}) but carries " + f"the cache-refresh-only guard, so it is skipped on push and " + f"the cache it owns goes stale on main" + ) + if name not in CACHE_REFRESH_JOBS and name not in guarded: + failures.append( + f"{CACHE_REFRESH_WORKFLOW}: job `{name}` has no " + f"`if: ${{{{ !inputs.cache-refresh-only }}}}`, so it runs on " + f"every push to main where the merge queue has already tested " + f"the same tree. Add the guard, or add the job to " + f"CACHE_REFRESH_JOBS with the cache entry it writes" + ) + + # The guards above do nothing unless the caller actually sets the input; + # its default is false, so a dropped `with:` block silently restores the + # full pipeline on push. + ci = (WORKFLOWS / "ci.yml").read_text(encoding="utf-8").splitlines() + if not any(CACHE_REFRESH_INPUT.match(line) for line in ci): + failures.append( + "ci.yml never passes `cache-refresh-only:` to pr_build_linux.yml. " + "The input defaults to false, so without it every push to main runs " + "the full pipeline again" + ) + + for failure in failures: + print(f"cache refresh scope: {failure}") + return not failures + + if __name__ == "__main__": ok = check_change_filters() ok = check_event_policy() and ok @@ -613,6 +709,7 @@ def check_required_checks(): ok = check_artifact_names() and ok ok = check_local_actions_have_checkout() and ok ok = check_required_checks() and ok + ok = check_cache_refresh_scope() and ok if not ok: sys.exit(1) print("CI config checks passed") diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index cf617d2e26..a66658b917 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -65,6 +65,12 @@ "!spark/src/test/scala/org/apache/spark/sql/benchmark/**", "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala", ], + # Same inputs as build_linux: not a separate job but a second POLICY + # decision for the same call, selecting the full pipeline rather than the + # cache-populating subset. ci.yml folds it into the reusable workflow's + # `cache-refresh-only` input. Populated below, after the dict, so the two + # lists cannot drift. + "build_linux_full": [], "build_macos": [ "native/**", "common/**", @@ -318,6 +324,7 @@ ], } FILTERS["spark_4_1_hive"] = FILTERS["spark_4_1"] +FILTERS["build_linux_full"] = FILTERS["build_linux"] # Which events may run each job, independent of the path filters above. # @@ -342,10 +349,21 @@ # own branch or on main, and nowhere else. The queue runs on a throwaway # gh-readonly-queue/* branch, so whatever it saves is deleted with that # branch. Without a push run, a Cargo.lock or pom.xml change would leave - # main's cargo-registry, Maven and TPC-H/TPC-DS caches stale forever, and - # every later pull request would pay the delta on top of the restore-keys - # prefix match. + # main's cargo-ci, cargo-debug, Maven and TPC-H/TPC-DS caches stale + # forever, and every later pull request would pay the delta on top of the + # restore-keys prefix match. + # + # On push that is the *only* thing it is for. The queue already tested the + # exact tree that landed, so re-running the lints and the 5x4 linux-test + # matrix there tests nothing, and they are 514 of the 587 runner-minutes a + # push run costs. The split below keeps the cache writers on push and moves + # everything else behind `build_linux_full`. "build_linux": ["pr", "queue", "push"], + # The lints and the test matrix inside pr_build_linux.yml. Deliberately no + # "push": ci.yml turns this output into the workflow's `cache-refresh-only` + # input, so dropping "push" here is what trims the push tier down to the + # jobs that write an actions/cache entry. See issue #5929. + "build_linux_full": ["pr", "queue"], # macOS runners are the scarcest capacity we have, and the Linux build # already covers rustfmt and the Rust/JVM compile on every PR. The label # is for a change that touches platform-specific code. diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index c7b6c87de6..c5348a622b 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -52,6 +52,14 @@ is the Linux build, which also runs on push so that the dependency caches on `ma pull request can only restore caches saved on its own branch or on `main`, and the queue's temporary branch takes its caches with it when it is deleted. +That push run is for the caches and nothing else, so it runs in **cache-refresh-only** mode: only +the four jobs that own a cache entry (the native CI build, the Rust tests, and the two TPC-H/TPC-DS +jobs, the last two stopping before their query passes), plus the short `Lint` job the native jobs +depend on. The lints and the Comet test matrix are skipped, which is the difference between 587 +runner-minutes a push and about 73. If you add a job to `pr_build_linux.yml`, give it +`if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache that `main` needs; +`dev/ci/check-ci-config.py` fails the build if you forget. + Spark 3.4 is the one suite in neither tier. [Spark 3.4 support is deprecated](../user-guide/latest/compatibility/spark-versions.md#spark-34), so its Spark SQL suite no longer gates a merge. It remains available on demand: apply the `run-spark-3.4-tests` label to run it against a pull request, or trigger `ci.yml` from the Actions @@ -202,8 +210,9 @@ does; the macOS job differs from Linux only in the platform. The umbrella workflow, the reusable workflows it calls, and the routing tables are checked by `dev/ci/check-ci-config.py`, which runs in preflight. It enforces that every job feeds `Required Checks`, that the required check name in `.asf.yaml` matches the job that publishes it, -that artifact names are unique per producer, and that the routing policy matches its test cases. -Run it locally before pushing a CI change: +that artifact names are unique per producer, that the routing policy matches its test cases, and +that every job in `pr_build_linux.yml` is either a cache writer or skipped on push. Run it locally +before pushing a CI change: ```sh python3 dev/ci/check-ci-config.py