Skip to content

build: enable Spark SQL tests for released Spark 4.2.0 #4950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
andygrove wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

build: enable Spark SQL tests for released Spark 4.2.0 #4950

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
382d065
build: Enable Spark SQL tests for Spark 4.2.0-preview4
andygrove May 4, 2026
45273c0
fix: set spark.version.short to 4.2 in 4.2.0-preview4 diff
andygrove May 4, 2026
a3dc08a
Merge apache/main into spark-4.2.0
andygrove Jul 16, 2026
a9d9df4
fix: split CometInternalRowShim per version for Spark 4.2
andygrove Jul 16, 2026
102a043
build: target released Spark 4.2.0 in the spark-4.2 profile
andygrove Jul 16, 2026
c1e57f9
test: add Spark SQL test diff for released Spark 4.2.0
andygrove Jul 16, 2026
b4adc19
test: regenerate Spark 4.2 plan-stability goldens against 4.2.0
andygrove Jul 16, 2026
5e1735e
ci: run Spark SQL tests for Spark 4.2
andygrove Jul 16, 2026
b79fbae
build: drop stale preview4 reference from spark-4.2 profile comment
andygrove Jul 16, 2026
51a0085
ci: allow run-spark-4.2-tests to gate the preflight job
andygrove Jul 16, 2026
db8b9a5
chore: bump spark-4.2 profile to the released 4.2.0
andygrove Jul 17, 2026
78049c5
chore: regenerate Spark 4.2 plan-stability golden files
andygrove Jul 17, 2026
3e5007f
fix: resolve Spark 4.2 test failures from the version bump
andygrove Jul 17, 2026
edc4385
Merge remote-tracking branch 'apache/main' into spark-4.2-final-bump
andygrove Jul 17, 2026
3740622
Merge remote-tracking branch 'apache/main' into spark-4.2.0
andygrove Jul 17, 2026
1effbee
Merge branch 'spark-4.2-final-bump' into spark-4.2.0
andygrove Jul 17, 2026
0936975
fix: fall back to Spark when native scan schema has unsupported types
andygrove Jul 17, 2026
5d70d83
test: resolve Spark 4.2 SQL test failures and file specific tracking …
andygrove Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ is a `workflow_call` reusable invoked from the umbrella.
(PR+push) (PR+push) (PR+push)
| | |
v v v
spark_3_4 / spark_4_1 iceberg_1_8 / 1_9
(push or PR + label) (push only)
spark_3_4 / spark_4_1 / spark_4_2 iceberg_1_8 / 1_9
(push or PR + label) (push only)

reusable workflows invoked via `uses:`:
pr_build_linux.yml spark_sql_test_reusable.yml
Expand All @@ -62,6 +62,7 @@ is a `workflow_call` reusable invoked from the umbrella.
| `spark_4_0` | PR or push, paths matched | Spark 4.0 sources |
| `spark_3_4` | push, **or** PR with `run-spark-3.4-tests` label | Spark 3.4 sources |
| `spark_4_1` | push, **or** PR with `run-spark-4.1-tests` label | Spark 4.1 sources |
| `spark_4_2` | push, **or** PR with `run-spark-4.2-tests` label | Spark 4.2 sources |
| `iceberg_1_10` | PR or push, paths matched | Iceberg sources |
| `iceberg_1_8` | push only | Iceberg sources |
| `iceberg_1_9` | push only | Iceberg sources |
Expand All @@ -86,14 +87,14 @@ umbrella doesn't watch, or operate independently of the rest of CI:

## Reusable workflows (called by `ci.yml`)

| File | Called from `ci.yml` job(s) |
| --------------------------------- | -------------------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` |
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` |
| File | Called from `ci.yml` job(s) |
| --------------------------------- | --------------------------------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` |

## Modifying path filters

Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
github.event.label.name == 'run-spark-3.4-tests' ||
github.event.label.name == 'run-spark-4.0-tests' ||
github.event.label.name == 'run-spark-4.1-tests' ||
github.event.label.name == 'run-spark-4.2-tests' ||
github.event.label.name == 'run-iceberg-tests'
steps:
- uses: actions/checkout@v7
Expand Down Expand Up @@ -114,6 +115,7 @@
spark_3_5: ${{ steps.compute.outputs.spark_3_5 }}
spark_4_0: ${{ steps.compute.outputs.spark_4_0 }}
spark_4_1: ${{ steps.compute.outputs.spark_4_1 }}
spark_4_2: ${{ steps.compute.outputs.spark_4_2 }}
iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }}
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
Expand Down Expand Up @@ -263,6 +265,23 @@
spark-full: '4.1.2'
java: 17

spark_4_2:
name: Spark SQL Tests (Spark 4.2)
needs: changes
# Main-only by default; PRs need the `run-spark-4.2-tests` label. Spark 4.2
# support is still experimental, so keep it off the default PR path.
if: |
needs.changes.outputs.spark_4_2 == 'true' &&
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'run-spark-4.2-tests')))
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '4.2'
spark-full: '4.2.0'
java: 17

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
Comment on lines +268 to +284
iceberg_1_8:
name: Iceberg Spark SQL Tests (Iceberg 1.8)
needs: changes
Expand Down
22 changes: 22 additions & 0 deletions dev/ci/compute-changes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,28 @@
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
],
"spark_4_2": [
"native/**/src/**",
"native/**/Cargo.toml",
"native/Cargo.lock",
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/spark-3.4/**",
"!spark/src/main/spark-3.5/**",
"!spark/src/main/spark-3.x/**",
"!spark/src/main/spark-4.0/**",
"!spark/src/main/spark-4.1/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
"spark/pom.xml",
"dev/diffs/4.2.0.diff",
"pom.xml",
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
],
"iceberg_1_8": [
"native/**/src/**",
"native/**/Cargo.toml",
Expand Down
Loading
Loading