Data, Spark: Add format model TCK coverage for vectorized reads (WIP)

[joyhaldar]

Adds a TCK subclass for Spark's vectorized ColumnarBatch read path.

TestSparkVectorizedFormatModel redefines FILE_FORMATS and FORMAT_AND_GENERATOR to skip Avro, which has no ColumnarBatch model registered. This required making DataGenerator, DataGenerators and DataGenerators.ALL public in the test package.

Three base class changes were needed:

• readOnly() - vectorized models register readers only, so write tests are skipped.
• assertRecordsEqual() - read tests built expectations via convertToEngineRecords, which can't work for a batch type
• Read assertions moved inside try-with-resources, columnar batches don't survive the reader closing

[pvary]

I don't really like this one.

Could we just re-define FILE_FORMATS, and FORMAT_AND_GENERATOR in TestSparkVectorizedFormatModel?

  private static final FileFormat[] FILE_FORMATS = {FileFormat.ORC, FileFormat.PARQUET};
  private static final List<Arguments> FORMAT_AND_GENERATOR

This relies on the fact, how junit works internally, but no change is required in all of the tests.

I'm open for other ideas as well, this is just the first one occured to me

[joyhaldar]

Thank you for the suggestion @pvary I tried it out and here's what I found:

Redefining FILE_FORMATS in the subclass works as you described, those tests drop AVRO with no base class changes.

Redefining FORMAT_AND_GENERATOR doesn't work as is, because it references DataGenerators, which is package-private in org.apache.iceberg.data while TestSparkVectorizedFormatModel is in org.apache.iceberg.spark.data.

I tried making DataGenerators, DataGenerators.ALL and the DataGenerator interface public, and with that both fields can be redefined in the subclass. All AVRO cases are gone and I could remove supportedFormats() and the per test assumptions entirely. TestSparkFormatModel and TestFlinkFormatModel still pass.

Would you like me to go with that, or would you rather avoid widening the visibility of those test classes?

[pvary]

I'm fine with making the DataGenerator and related stuff public in the test packages

[joyhaldar]

Thank you @pvary.

1. Made DataGenerator, DataGenerators and DataGenerators.ALL public in the test package.
2. TestSparkVectorizedFormatModel redefines both FILE_FORMATS and FORMAT_AND_GENERATOR.
3. Dropped supportedFormats() and all the per test assumeSupportedFormat() calls from the base class.

[joyhaldar]

One thing I hit while doing this. When the subclass redefines FORMAT_AND_GENERATOR, it also has to repeat every filter the base applies. #17617 added supportsGenerator yesterday, my subclass didn't have it, and ORC + Variant came back and failed CI.

To fix it I'd have to either make supportsGenerator public, or copy the check into the subclass. Neither seems clean, and any new filter will have the same problem IIUC.

[pvary]

protected is enough

[pvary]

Shall we split the base test class to read-tests, and read+write-tests?
Using this method in all test seem awkward a bit.

If we decide so, we might want to do it in a different PR

[joyhaldar]

I agree with you, would love your advice.

Two ways we could do it:

1. Keep the read tests and shared helpers in BaseFormatModelTests, move the write tests into a BaseReadWriteFormatModelTests that extends it. Existing tests move to the new class.
2. Insert a BaseReadFormatModelTests above the current BaseFormatModelTests and put the read tests there. No existing subclass changes its extends.

And TestSparkVectorizedFormatModel would only extend the read-only class.

Would you like to do it in a follow-up PR or this PR?

[pvary]

Please separate out the test to:

• ReadFormatModelTests as a base, and move the read tests there
• BaseFormatModelTests inheriting from ReadFormatModelTests

I would do this way

[pvary]

This should be in a PR before this change

[pvary]

package private