Skip to content

Spark: Support nested geospatial values in StructInternalRow - #17741

Open
charliec05 wants to merge 1 commit into
apache:mainfrom
charliec05:agent/spark-geospatial-collections
Open

Spark: Support nested geospatial values in StructInternalRow#17741
charliec05 wants to merge 1 commit into
apache:mainfrom
charliec05:agent/spark-geospatial-collections

Conversation

@charliec05

Copy link
Copy Markdown

Closes #17720

What changed

  • Add robust conversion for nested GEOMETRY / GEOGRAPHY values in StructInternalRow, including list/map/struct collections for Spark 4.1.
  • Handle WKB conversion via helper that supports both byte[] and ByteBuffer inputs and preserves correct SRID for GEOMETRY values.
  • Add TestStructInternalRowGeospatial covering scalar and nested collection conversions, null handling, and byte[]/ByteBuffer inputs.

Testing

  • ./gradlew --no-parallel --max-workers=2 -DsparkVersions=4.1 :iceberg-spark:iceberg-spark-4.1_2.13:test --tests org.apache.iceberg.spark.source.TestStructInternalRowGeospatial
  • ./gradlew --no-parallel --max-workers=2 -DsparkVersions=4.1 :iceberg-spark:iceberg-spark-4.1_2.13:spotlessCheck

Notes

@github-actions github-actions Bot added the spark label Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark: Support nested geospatial values in StructInternalRow for older versions

1 participant