Support Spark 4.3

[pan3793]

Why are the changes needed?

Spark 4.3.0 is currently in the RC voting phase. This prepares Kyuubi for Spark 4.3 and validates it against the RC artifacts.

What changes are proposed?

• Adds the spark-4.3 profile and kyuubi-extension-spark-4-3 module.
• Adapts the Spark extension, Hive connector tests, and Arrow runtime for Spark 4.3 API changes.
• Adds Spark 4.3 CI, release, distribution, codecov, formatting, and documentation coverage.
• Lakehouse engines still fall back to Spark 4.1 artifacts until native Spark 4.3 support is available.
• The temporary Spark staging repository and binary archive reference v4.3.0-rc1; they must be updated if the RC changes or after the final release is published.

How was this patch tested?

• build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-3 -am -Pspark-4.3 -Pscala-2.13
• build/mvn test -pl extensions/spark/kyuubi-spark-connector-hive -am -Pspark-4.3 -Pscala-2.13
• build/mvn test -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am -Pspark-4.3 -Pscala-2.13
• build/mvn test -pl externals/kyuubi-spark-sql-engine -Pspark-4.3 -Pscala-2.13
• ./dev/reformat

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: Codex

[wangzhigang1999]

I found a Spark 4.3 compatibility gap while testing this PR through Kyuubi JDBC on Kubernetes.

Open a session with:

jdbc:hive2://<host>:10009/default;#spark.sql.timestampNanosTypes.enabled=true

Then run:

SELECT localtimestamp(9);

Spark finishes the query, but Beeline fails while fetching the result-set metadata:

IllegalArgumentException: Unrecognized type name: timestamp_ntz(9)
  at SchemaHelper.toTTypeId(SchemaHelper.scala:59)

SchemaHelper currently handles TimestampType and TimestampNTZType. Spark 4.3's TimestampNTZNanosType and TimestampLTZNanosType extend AnyTimestampNanoType, so they reach the fallback case.

Could we add handling and tests for both nanos types at precisions 7–9, including metadata precision and Thrift/Arrow result conversion?

[wangzhigang1999]

I tested this PR at a0c81c9c49677ce005993847554529cb9395af15 on ACK using Spark 4.3.0 RC1 and Scala 2.13. I focused on Kyuubi's Spark 4.3 compatibility paths; CI covers the unit tests.

Area	Coverage	Result
Build and packaging	Spark 4.3 distribution and Kyuubi image	Passed
Spark SQL Engine	JDBC sessions and Kubernetes Driver/Executor	Passed
Spark 4.3 extension	Module loading, parser, dynamic shuffle, Watchdog, and Z-order smoke tests	Passed
Arrow compatibility	Spark 4.3 toArrowSchema signature, none/zstd, and nested types	Passed
Spark 4.3 SQL compatibility	ASOF JOIN, map DISTINCT, zero-length Parquet, column comment removal, and new unix_* inputs	Passed
Nanosecond timestamps	NTZ/LTZ precisions 7, 8, and 9 through JDBC and Arrow	Failed

With spark.sql.timestampNanosTypes.enabled=true, all six nanosecond timestamp variants fail during Kyuubi result metadata conversion:

Unrecognized type name: timestamp_{ntz|ltz}(7|8|9)

This was the only Kyuubi compatibility issue found in this ACK validation. Since nanosecond timestamps are disabled by default, this could either be fixed in this PR or documented as a known Spark 4.3 limitation.

[pan3793]

@wangzhigang1999, thanks for the additional testing. for TimestampLTZNanosType, I suggest adding support for it in a dedicated PR, and I guess we are also missing some other new data types that spark added recently.

if you want to work on this eagerly, we can merge this PR without waiting for Spark 4.3.0 GA, thought?