Skip to content

[CI] Restore Java 17 in the Spark Versions PreCommit so the Spark 4 module runs - #40093

Merged
Abacn merged 1 commit into
apache:masterfrom
tkaymak:ci-spark-versions-java17
Sep 10, 2026
Merged

[CI] Restore Java 17 in the Spark Versions PreCommit so the Spark 4 module runs#40093
Abacn merged 1 commit into
apache:masterfrom
tkaymak:ci-spark-versions-java17

Conversation

@tkaymak

@tkaymak tkaymak commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

beam_PreCommit_Java_Spark_Versions runs :runners:spark:3:sparkVersionsTest :runners:spark:4:build -Pjava17Home=$JAVA_HOME_17_X64. #39205 changed its Java setup from 17 plus 11 to 11 only, so JAVA_HOME_17_X64 is empty, java17Home resolves to nothing, and BeamModulePlugin disables every task of runners/spark/4 because its required JDK is missing. Every run since then prints

> Task :runners:spark:4:compileJava SKIPPED
> Task :runners:spark:4:test SKIPPED
> Task :runners:spark:4:build SKIPPED
BUILD SUCCESSFUL

and reports green. The Spark 4 unit tests, including the streaming source tests from #39971 and the translator tests in #40090, have not executed in CI in that period. beam_PreCommit_Java compiles the module on its native JDK but does not run its tests.

This restores the two version setup that #38453 introduced, same pattern as beam_PreCommit_SQL_Java17. Java 11 stays the default JDK for the Spark 3 part of the job, Java 17 is handed to the Spark 4 module through java17Home.

Verification after merge: the next master run of the workflow must show :runners:spark:4:test executed. Local equivalent, ./gradlew :runners:spark:4:build -Pjava17Home=/path/to/jdk17 runs 234 tests.

R: @Abacn

…odule runs

apache#39205 replaced the two version Java setup with Java 11 only. The job still
passes -Pjava17Home=$JAVA_HOME_17_X64, which is empty without the second
JDK, and the build plugin disables every task of runners/spark/4 when its
JDK is missing. Since then :runners:spark:4:build has been SKIPPED in every
run and the module's tests have not executed in CI.
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Amar3tto for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Abacn
Abacn merged commit 33dc11c into apache:master Sep 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants