chore: bump spark-4.2 profile to the released 4.2.0#4960
Merged
Conversation
Spark 4.2.0 is now in Maven Central, so the spark-4.2 profile no longer needs to target a preview. Point spark.version at 4.2.0 and make the main sources compile against it. - Drop the WIP compile-only comment on the spark-4.2 profile. - Spark 4.2.0 bundles its own copy of org.apache.datasketches.memory.internal.ResourceImpl inside spark-catalyst, colliding with the transitive datasketches-memory jar and failing the BanDuplicateClasses enforcer rule; ignore that class. - Spark 4.2 removes GeographyVal / GeometryVal and replaces getGeography / getGeometry on SpecializedGetters with a single getBinaryView returning BinaryView. CometInternalRowShim can no longer be shared from the spark-4.1+ source root, so split it into per-version spark-4.1 and spark-4.2 copies. - Correct the stale Jetty comment that referenced preview4. This does not wire up the Spark SQL tests for 4.2; it only moves the profile onto the released version and keeps it compiling.
Regenerated with dev/regenerate-golden-files.sh --spark-version 4.2 against the released 4.2.0. The q2, q5, q54 (v1_4) and q5a (v2_7) goldens are pruned because their plans now match the shared fallback on 4.2.0, and q77a (v2_7) is added to record a degraded plan.
parthchandra
approved these changes
Jul 17, 2026
parthchandra
left a comment
parthchandra
left a comment
Contributor
There was a problem hiding this comment.
lgtm. Thanks @andygrove
Two Spark 4.2.0 changes surfaced by moving the profile off preview4: Iceberg scans job: Spark 4.2.0 turned `connector.catalog.View` from an interface into a class. No Iceberg spark-runtime is published for 4.2, so the build reuses the 4.0 runtime, whose `SparkView implements View` now throws IncompatibleClassChangeError at class-load and aborts the Iceberg suites. Report Iceberg as unavailable on Spark 4.2 in the shared probes so the suites skip. Guard the fuzz suite's beforeAll with an early return (cancelling from beforeAll aborts the suite) and add the missing `assume(icebergAvailable)` to one native-scan test. Expressions job: Spark 4.2 normalizes NaN / -0.0 for array_distinct and the array set operations by wrapping their inputs as `KnownFloatingPointNormalized(ArrayTransform(arr, x -> NormalizeNaNAndZero(x)))`. The serde only handled a scalar `NormalizeNaNAndZero` child and fell back. Since `KnownFloatingPointNormalized` is a runtime no-op tag, serialize any other child directly and let its serde (the ArrayTransform codegen dispatcher) carry the normalization, keeping these operations native.
# Conflicts: # spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
Contributor
|
Great to see! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #4142.
Rationale for this change
Spark 4.2.0 is now available in Maven Central, so the
spark-4.2profile no longer has to target a preview. This PR moves the profile onto the released version and makes the main sources compile against it, without yet wiring up Spark's own SQL tests for 4.2. It is a smaller slice of the larger bring-up in #4950, intended to land the version bump on its own.Spark 4.2.0 is not simply
4.2.0-preview4with the qualifier dropped: it changesSpecializedGettersand repackages a dependency, so a couple of real reconciliations are needed to compile.What changes are included in this PR?
Build
spark.versionmoves from4.2.0-preview4to4.2.0, and the WIP compile-only comment on thespark-4.2profile is dropped.org.apache.datasketches.memory.internal.ResourceImplinsidespark-catalyst, colliding with the transitivedatasketches-memoryjar and failing theBanDuplicateClassesenforcer rule. That class is now ignored.Shims
GeographyVal/GeometryValand replacesgetGeography/getGeometryonSpecializedGetterswith a singlegetBinaryViewreturningBinaryView.CometInternalRowShimcan no longer be shared from thespark-4.1+source root, because the major, minor-plus and minor shim directories are all compile source roots (aspark-4.2copy would be a duplicate class, not an override). It is split into per-versionspark-4.1andspark-4.2copies.Plan stability
dev/regenerate-golden-files.sh --spark-version 4.2against the released 4.2.0. Theq2,q5,q54(v1_4) andq5a(v2_7) goldens are pruned because their plans now match the shared fallback on 4.2.0, andq77a(v2_7) is added to record a degraded plan.This PR intentionally does not enable the Spark SQL tests or add CI jobs for 4.2; that work is tracked separately in #4950.
How are these changes tested?
./mvnw -Pspark-4.2 -DskipTests test-compilebuilds the main and test sources against the released Spark 4.2.0../mvnw -Pspark-4.1 -DskipTests test-compilestill passes, confirming the shim split did not regress the 4.1 profile. The plan-stability suites (CometTPCDSV1_4_PlanStabilitySuiteandCometTPCDSV2_7_PlanStabilitySuite) pass on the regenerated goldens.