Spark: Read variant columns through the row reader to avoid a shredded vectorization failure

[nssalian]

Rationale for this change

In my testing, I found that Vectorized Parquet reads throw UnsupportedOperationException when they hit a shredded variant column (typed_value subtree). SparkBatch tried to route shredded files away from the vectorized reader using the manifest variant bounds and the write.parquet.shred-variants property, but neither is sound. Bounds are optional (a file written with metrics none/counts carries none), and the property reflects writer intent, not how existing files were written. Shredding can also be enabled through the session conf or a write option with the table property left false. A shredded file can therefore reach the vectorized reader and crash, and it cannot be detected soundly at plan time, since shredding is a per-file Parquet-footer fact that is not recorded in the manifest.

After checking all possible options, this seemed like the easiest to do to avoid any edge cases. Follow up section has more.

Changes

Spark 4.0 and 4.1: Route variant columns to the row reader in the batch path for now. SparkBatch.supportsParquetBatchReads now returns false for any variant column, and the unsound bounds and property/metrics checks are removed. A query that projects a variant column reads through the row reader, which reconstructs both shredded and unshredded variants correctly. Queries that do not project a variant column are unaffected and still vectorize.

Also, SparkScanBuilder no longer force-loads column stats for variant columns on read, since that plumbing existed only to feed the removed bounds check.

Testing

Added testReadShreddedViaSessionConfWithoutTableProperty (v4.0 and v4.1). It enables shredding through the session conf with the table property left false and metrics disabled, then reads under vectorization and asserts the variant values are correct. On the previous routing the shredded file reached the vectorized reader and threw UnsupportedOperationException, so the test failed; with this change it reads through the row reader and passes. It guards against the crash returning: any routing that lets a shredded file reach the vectorized reader again fails the read, and the test.

Follow up

This turns off vectorized reads for variant columns, including unshredded ones, since plan time cannot tell them apart. The follow-up is to write the vectorized reader to reconstruct shredded variants, which lets variant vectorize again for both shredded and unshredded data in a subsequent release. This is a more involved work that will take a while so disabling this for now until that can go in. I'll open the fix in parts soon.

[nssalian]

Failures seems from maven, needs a re-run