Databricks + DBT + Sedona compatibility

[alexeyegorov]

At our startup, we use the combination of Sedona and DBT on Databricks. However, the compatibility of latest Sedona versions with Databricks Runtime is not easy or clear to understand.

Following the documentation I though 1.7.2 and above should be compatible with DBR 16.4 LTS. I was able to run the Sedona spatial functions, but I could not the resulting dataset as Delta file. Using DBT I ran into a following error:

I have tried the following combinations:

• Sedona 1.7.2 + DBR 16.4 (above error)
• Sedona 1.8.0 + DBR 16.4 (above error)
• Sedona 1.8.0 + DBR 17.X (some Spark 4.0 incompatibility)

Does anyone here have experience with the setup? If there is something in development that we might test using our setup, I would be open to support that.

Until then, it seems that the Sedona 1.7.0 and DBR 15.4 is just the one stable solution.

[github-actions]

Thank you for your interest in Apache Sedona! We appreciate you opening your first issue. Contributions like yours help make Apache Sedona better.

[jiayuasu]

@alexeyegorov DeltaLake never officially supports Geometry data type of Sedona. So even if something that happens to work before, there is no guarantee that it will work in the future. Especially given that the Databricks DeltaLake is not 100% identical to OSS DeltaLake, we can only speculate.

I would recommend one of the following 2 options

1. Read / write geometries via WKB Binary data type in DeltaLake.
2. Read / write geometries via geoparquet data source using Sedona 1.8.0. You will get native geo type experience and spatial filter pushdown too.

DBR 17.X is known to have issues with Sedona 1.8.0 and we are waiting for a LTS version of DBR 17 to fix it.

[alexeyegorov]

Thanks for your reply @jiayuasu

We were expecting that Sedona will add support for Delta's new geometry/geography types with the newest release? Or is it possible to achieve this using Iceberg? In a later case, we could hope that Databricks allows users not only to create the metadata for other formats, but to simply write Iceberg tables.

In case we would need to switch to read/write WKB types, that would mean rewriting a lot of code. But it would be possible though while taking a lot of convenience.