Spark 4.1: Test geometry and geography DML and fall back from vectorized reads#17149
Open
huan233usc wants to merge 4 commits into
Open
Spark 4.1: Test geometry and geography DML and fall back from vectorized reads#17149huan233usc wants to merge 4 commits into
huan233usc wants to merge 4 commits into
Conversation
…zed reads Add end-to-end DELETE/UPDATE/MERGE tests for geometry and geography columns under merge-on-read at format version 3, covering deletion vectors, nested geo in a struct, and null values. Geometry and geography are primitive types, so they slipped through the vectorized Parquet read gate in SparkBatch even though there is no Arrow geo vector yet, which threw when scanning geo columns with vectorization enabled. Exclude the two geo types so they fall back to the non-vectorized reader.
added 3 commits
July 9, 2026 14:47
Select the delete snapshot by operation instead of the latest committed_at, whose millisecond granularity could tie with the insert snapshot, and give the test its own table so there is exactly one delete snapshot to inspect.
Copy-on-write is the default row-level mode, and its rewrite path reads geo values back out of the surviving data files, so add DELETE and UPDATE tests that exercise it (the merge-on-read tests only cover deletion vectors).
Fold the duplicated merge-on-read and copy-on-write DELETE and UPDATE variants into single tests parametrized over the write mode, and run MERGE and the nested delete over both modes too. The deletion-vector assertion applies only to merge-on-read.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds end-to-end Spark SQL DML tests for
geometryandgeographycolumns, andfixes a vectorized-read bug those tests surfaced. This completes the Phase-1
geo end-to-end item (row-level DML with deletion vectors / nested geo / nulls)
and builds on #17073 (Spark geo value path) and #16982 (Parquet geo WKB values).
Tests
TestSparkGeospatialpreviously covered onlyINSERT→SELECTread-back. Thisadds row-level DML on a geo table under merge-on-read + format version 3
(deletion vectors):
testDeleteGeospatialMergeOnRead—DELETEleaves a survivor in a single datafile, and the snapshot summary confirms a deletion vector was written
(
added-dvs = 1); the surviving null-geo row round-trips.testUpdateGeospatialMergeOnRead—UPDATEswaps a row's geometry; othersunchanged.
testMergeGeospatialMergeOnRead—MERGEupdates a matched row and inserts anot-matched row, both carrying geo values.
testDeleteNestedGeometryMergeOnRead— geometry nested in aSTRUCT; delete bya nested non-geo field, surviving nested geometry (and a null nested geometry)
still round-trip.
Geo values are asserted via
hex(st_asbinary(...))round-trips and DML filters onnon-geo columns, since Spark 4.1 has no spatial predicates.
Vectorization fallback (production fix)
geometry/geographyare primitive types, so they passedSparkBatch.supportsParquetBatchReads(NestedField)and were routed to thevectorized Parquet reader — but there is no Arrow geo vector yet, so scanning a geo
column with vectorization enabled threw
UnsupportedOperationException: Unsupported primitive type: geometry. This excludesthe two geo types from vectorized reads so they fall back to the row-based reader.
With this fix,
testGeospatialWkbReadBacknow runs (no longer skipped) withvectorization both off and on, proving the fallback. A dedicated Arrow geo
vector remains future work.