ci: share one Linux native build across CI workflows

[sunchao]

Which issue does this PR close?

Part of #5830. Follow-up to #3249, which shared native artifacts within individual workflows.

Rationale for this change

The Linux, Spark SQL, and Iceberg workflows compile the same default Linux native library separately. Under the current routing policy, a shared source change selects three native-consuming callers in the merge queue, six in the nightly run, and nine on manual dispatch. Build the library once and share it with those callers to reduce duplicate compilation. Actual time savings depend on cache state and still need measurement in CI.

Splitting the Linux workflow must also preserve the cheaper push-to-main mode: refresh the Cargo, Maven, and TPC dataset caches without repeating Java/Scala lint, compilation checks, Celeborn checks, or JVM/query tests that already ran.

What changes are included in this PR?

• Add one shared Linux native producer with the existing Cargo ci profile, JDK 17, compiler flags, artifact retries, and one-day retention. All nine native-consuming callers depend on it, and its result feeds Required Checks.
• Move the six independent Linux checks into a reusable workflow that runs alongside the producer. Preserve their Maven bootstrap steps, JDK 17 settings, and Rust formatting prerequisite. Spark JVM compilation uses setup-java without Rust setup; Spark/JDK-specific JVM artifacts remain separate.
• Derive native selection from all routes for each consumer, including Spark 4.1 core/Hive and Linux default/all-profile outputs. Preserve the current PR, queue, nightly, label, and manual policy, including the independent Delta/PyArrow workflows.
• Pass identical cache-refresh-only and profiles inputs to both Linux callers. On main pushes, retain the native producer, Rust formatting/debug tests, and both TPC cache-population jobs. Nightly runs skip previously completed checks and TPC work. Cargo CI/debug caches are saved only on push.
• Keep Linux profile selection in a small prepare-matrix job after moving lint. Preserve the default, nightly, and all-profile matrices.
• Extend configuration guards across all three Linux workflows and check each caller's mode inputs separately. Mutation tests cover lost guards, disabled/missing cache writers, missing inputs, profile-matrix wiring, native selection, artifact dependencies, and Maven bootstrap.
• Update the workflow documentation and contributor guides to reflect the moved jobs and cache ownership.

How are these changes tested?

Rebased onto main at 343143fb9. Independent comparison verified the moved job definitions, consumer test steps and suite matrices, all nine artifact dependencies, Required Checks coverage, and profile selection (1/4/5 rows for PR/nightly/all).

Passed locally on 3c31e5aa8:

• CI configuration checker and 41 configuration regression tests.
• 23 native-selection tests, including 53,248 path-mask/event combinations and real CLI coverage for cache refresh, nightly, profile labels, and Hive-only labels.
• Actionlint 1.7.12 with ShellCheck disabled, matching preflight.
• Apache RAT, suite inventory, benchmark inventory (49 suites), Iceberg shard tests (15), and PR-label tests (4).
• Markdown formatting, whitespace checks, and rebase range-diff review.

Fresh hosted validation is pending: commit CI and Spark 4.1 opt-in CI. The existing four opt-in labels remain applied; the additional Spark 4.1 label requests the ninth consumer now that it is queue-only by default.

No native or Spark runtime build was run locally. Hosted push/nightly execution and failed-job reruns after artifact expiry remain unverified; after the shared artifact expires, rerun the full workflow.

[rich7420]

LGTM

[andygrove]

LGTM but I would like to merge #5842 first so we can unblock setting up the merge queue to reduce CI resource usage. I reviewed this PR assuming that it will go in after #5842.

Once #5842 is in, two things will need to happen on the rebase here:

1. ci.yml merges cleanly between the two PRs, but #5842 adds a required_checks aggregator whose needs: must list every ci.yml job, and a preflight guard in check-ci-config.py that enforces it. The merged file fails that guard until pr_build_linux_checks and build_linux_native are added to required_checks.needs. The guard message names them, so the rebase will tell you. build_linux_native in particular has to be there and cannot be exempted: if the producer fails, GitHub marks all nine consumers skipped, and the aggregator treats skipped as pass, so without the producer in needs a broken native build would produce a green Required Checks.

2. #5842 hardens Lint Scala (syntactic) against Maven Central connection resets by splitting it into a retried cs launch scalafix:0.14.6 -- --version warm-up followed by the real check under cs launch --mode offline. Since this PR moves that job into pr_build_linux_checks.yml, that split needs to come along. The iceberg_spark_test_reusable.yml change in #5842 (routing the shard-inventory upload through upload-artifact-retry) merges cleanly onto your prepare-shards version.

After both land, check-ci-config.py will carry two hand-rolled YAML readers (block_mapping here and the line regexes in #5842), both justified by PyYAML not being on the preflight runner. I will open a follow-up issue to install PyYAML in preflight and collapse them rather than leave both in place.

I verified the two-way compatibility locally: your checker passes against a ci.yml that includes the required_checks job, and the #5842 checker passes against your ci.yml once the two ids are added.

[andygrove]

One more heads-up from the merge-queue side, following on from my review above.

#5843 (the PR that actually enables the queue, stacked on #5842) changes POLICY in dev/ci/compute-changes.py so that a push to main selects only docs. Every test job moves to the merge_group event instead, because the queue has already tested the exact tree that lands.

That will break test_main_and_manual_runs_include_legacy_consumers in dev/ci/test-native-build-selection.py here, which asserts event="push" selects every native consumer. Whichever of #5841 and #5843 lands second needs to update that test (the workflow_dispatch half stays true). If this PR lands first, #5843 will pick it up on its rebase along with adding pr_build_linux_checks and build_linux_native to required_checks.needs. If #5843 lands first, the push assertion here should become merge_group.

Also FYI, run-spark-3.4-tests and run-spark-3.5-tests now exist as labels, so the label-gated selection tests here have real labels behind them.

[andygrove]

Three things, all about the rebase rather than the design. The producer and consumer wiring itself looks right to me, and I checked that build_linux_native is the exact union of the nine consumer outputs on this head.

#5852 landed on main after this branch was last pushed, adding a Bootstrap Maven step calling ./.github/actions/maven-bootstrap to five jobs in pr_build_linux.yml. Three of those five are the jobs this PR moves into pr_build_linux_checks.yml, namely lint-java, build-spark-4-1 and celeborn-reflection-compatibility. A rebase cannot carry that step into a file this branch creates from scratch, and pr_build_linux.yml conflicts, so the natural resolution is to take the deletion and lose the retry for those three. The two TPC jobs stay behind and keep theirs. Nothing in check-ci-config.py verifies that a job calling a bare ./mvnw has the bootstrap step ahead of it, so this would be silent until the next Maven Central blip takes out a queue-gating job. Could you add it back to those three after the rebase, and update the ROUTING_CASES comment that says the composite is called only from pr_build_linux.yml? Given how easily it drops out in a file move, is it worth a guard for it?

NATIVE_CONSUMERS is a flat list of output keys, one per consumer job, and #5871 has already broken that assumption on main. It added spark_4_1_hive as a second output feeding the same spark_4_1 job, which is now gated on needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'. On a labeled run with run-spark-4.1-hive-tests, spark_4_1 is false and spark_4_1_hive is true, so the union computes build_linux_native as false, the producer is skipped, and spark_4_1 is then skipped through its needs even though its if is true. The label would quietly do nothing and Required Checks would still go green. compute-changes.py is the one file that auto-merges cleanly here, so the union will miss the new key silently. native_selection_failures will fail preflight on the if: string mismatch, but the tempting fix there is to relax the comparison, which leaves the hole. Would it make sense to let a consumer map to a set of output keys rather than one?

Last one is coverage. The hosted run only selected pr_build_linux, spark_4_1 and iceberg_1_11, so six of the nine rewired consumers have not run against the shared artifact. spark_3_4 is the one I would most want to see, because its build job in spark_sql_test_reusable.yml is the only consumer whose toolchain setup actually changed, swapping ./.github/actions/setup-builder for a bare actions/setup-java@v4 at JDK 11, and it now consumes a libcomet.so linked against JDK 17. The linux-test Spark 3.4 lane already proves the library loads under JDK 11, but the new setup path has no coverage. These are queue-tier jobs, so a break blocks every merge rather than one PR. Could you apply run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests and run-iceberg-tests once on the rebased head?

[sunchao]

@andygrove, addressed the three items from your latest review and rebased onto main at 451c999632; the PR is now at 79c161b293 with no merge conflicts.

1. Maven bootstrap: preserved the bootstrap in all three moved jobs and both TPC jobs, and corrected the routing comment. The new guard requires an earlier unconditional bootstrap with failures propagated before direct ./mvnw commands in both Linux workflows. Regression mutations remove, move, condition, or ignore the bootstrap's failure in each of the five jobs.
2. Hive routing: NATIVE_CONSUMERS now maps caller IDs to all selecting outputs. spark_4_1 includes both core and Hive, so a Hive-only label starts the native producer. Exact caller, OR-gate, export, and dependency validation remains enforced. Compute and CLI tests cover core-only, Hive-only, both, queue, and manual runs; upstream's module selection is preserved.
3. Coverage: applied all four requested labels after pushing this head: run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests, and run-iceberg-tests. The regular PR run and four additional opt-in runs (1, 2, 3, 4) are pending. This requests coverage of all nine rewired consumers, including the Spark 3.4/JDK 11 setup path; runtime results are not yet confirmed.

Local validation passed: 36 configuration regression tests, 21 native-selection tests (299,008 combinations), actionlint, suite/benchmark inventory checks, 15 Iceberg shard tests, 4 PR-label tests, Markdown formatting, and whitespace checks. The PR description now reflects the implementation and pending hosted coverage.

[andygrove]

Happy to approve once the following issues are resolved.

Thanks for the quick turnaround on the last round. All three items are addressed: the Maven bootstrap is back in the three moved jobs and guarded, NATIVE_CONSUMERS maps callers to tuples so the Hive-only label starts the producer, and all four label runs came back green, so every one of the nine consumers has now run against the shared artifact, including the Spark 3.4 setup-java-only build path. I also ran the three Python checks locally on 79c161b and they pass in about five seconds total, so the preflight cost is fine.

1. Rebase hazard from #5897. chore: drop support for JDK 11 landed on main after this head was pushed. It edits the lint-java matrix in pr_build_linux.yml (the Spark 3.4 entry becomes JDK 17 and JAVA_TOOL_OPTIONS drops the version conditional), and flips java: 11 to 17 for spark_3_4 and iceberg_1_8 in ci.yml and for preflight. The ci.yml and linux-test hunks should auto-merge, but the lint-java hunk conflicts with this PR's deletion of that block, and pr_build_linux_checks.yml is a new file, so the natural resolution drops the change and the new file keeps JDK 11. Same shape as the bootstrap step last time. It fails loudly this time because the new Maven enforcer rejects JDK 11, so no guard is needed, but could you carry those edits into pr_build_linux_checks.yml on the rebase?

2. Contributor doc pointer. docs/source/contributor-guide/adding_a_new_spark_version.md (around line 112) tells authors to add the compile-only job to pr_build_linux.yml, but build-spark-4-1 now lives in pr_build_linux_checks.yml. Could you update the file name there? The suite-matrix instructions in development.md and check-suites.py still point at pr_build_linux.yml correctly since linux-test stays put.

Nothing else from me. The wiring, the main-only cache write on the single producer, artifact retention, the permissions blocks, and the README rewrite all look right.

[sunchao]

@andygrove, both items from your latest review are addressed. Rebased onto main at 39da85081; the new head is a126925b7 and GitHub reports no merge conflicts.

1. JDK 17: carried the Spark 3.4/JDK 17 lint matrix entry and unconditional JAVA_TOOL_OPTIONS into pr_build_linux_checks.yml. The preflight, Spark 3.4, Iceberg 1.8, and Linux test updates from chore: drop support for JDK 11 #5897 are also preserved. All six moved jobs match the current upstream jobs.
2. Contributor guide: the compile-only instructions now point to pr_build_linux_checks.yml. The suite-matrix instructions still point to pr_build_linux.yml, where linux-test remains.

The rebase also preserves Spark 3.4's new label/manual-only policy. Updated the selection regression to verify that queue runs select the other eight consumers, that a Spark 3.4-only queue change does not start native compilation, and that labels cannot opt Spark 3.4 back into a queue run. Its label and manual paths remain covered through both Python and the CLI.

Local checks passed: 36 configuration tests, 21 selection tests (including 299,008 combinations), actionlint, suite/benchmark checks, 15 Iceberg shard tests, 4 PR-label tests, Markdown formatting, and whitespace checks. Updated the PR description with the previous green coverage and the fresh CI run. The four opt-in labels remain applied; results on this rebased head are pending.

[andygrove]

One rebase item, and this time only half of it is loud.

#5930 landed on main after this head. It adds a cache-refresh-only input to pr_build_linux.yml so that push to main runs only the jobs that own an actions/cache entry, and check-ci-config.py grew a sixth invariant pinning which jobs survive it. Three of the five jobs it names, lint, build-native and linux-test-rust, are jobs this PR moves into the two new workflows.

I merged main into a126925b7 locally. Four files conflict and compute-changes.py auto-merges correctly, the FILTERS["build_linux_full"] = FILTERS["build_linux"] alias picks up your two new paths for free. Preflight then fails on the three stale CACHE_REFRESH_JOBS names, on linux-test having lost its guard in the move, and on two ROUTING_CASES entries needing build_linux_full. All of that is loud and the messages say what to do.

The silent part is what is left after you do it. I dropped the three stale names, put the guard back on linux-test, and check-ci-config.py prints CI config checks passed. Both of your test suites pass too, 21 and 36. But pr_build_linux_checks.yml is gated on build_linux, which has a push tier, and it has no cache-refresh-only input, so the whole thing runs on every push to main. I confirmed the selection with EVENT_NAME=push python3 dev/ci/compute-changes.py on the merged tree: build_linux=true, build_linux_full=false. Using this PR's own durations that is lint-java x4 at 19m10s, Celeborn x2 at 12m03s, build-spark-4-1 at 4m21s and scalafix-syntactic at 25s, about 36 runner-minutes a push. #5930 took the push tier from 587 to about 73, so this puts it back to roughly 109.

Could you carry the input into pr_build_linux_checks.yml on the rebase, wired from ci.yml the same way as pr_build_linux with ${{ needs.changes.outputs.build_linux_full != 'true' }}, and guard those four jobs? lint and linux-test-rust should stay unguarded, since linux-test-rust owns the cargo-debug cache and lint gates it, which is the same reasoning #5930 used for the originals. Passing an input does not change the caller's if:, so your linux_checks_failures invariant that the two callers share a condition still holds.

The guard itself also needs to become multi-workflow. CACHE_REFRESH_WORKFLOW is a single Path today, and once the cache writers live in three files a table of {workflow: {job: reason}} is what actually expresses the invariant. Would it be worth extending the CACHE_REFRESH_INPUT check the same way, so a caller that forgets the input for either workflow fails preflight rather than only the pr_build_linux one? Your test-ci-config.py mutation harness is the natural place to pin it, a mutation that drops the guard from a job in pr_build_linux_checks.yml would have caught exactly what I hit.

Two smaller pieces of the same rebase. docs/source/contributor-guide/ci.md lines 55-60 and 214 name the five cache-writing jobs and tell contributors to add the guard when they add a job to pr_build_linux.yml. That file is not in this PR and merges clean, so it goes stale without saying anything. And the build_linux comment in POLICY still says the push tier is there for "main's cargo-ci, cargo-debug, Maven and TPC-H/TPC-DS caches", but cargo-ci now lives in build_linux_native.yml and cargo-debug in pr_build_linux_checks.yml, so that one output is load-bearing for three workflows now rather than one. Worth spelling out, because it is the thing that keeps the shared producer running on push at all.

Nothing else from me. The producer and consumer wiring, the union over NATIVE_CONSUMERS, the Maven bootstrap guard and the JDK 17 carry-over all look right on this head, and all nine consumers came back green.