[core][spark] Support selected-key pushdown for shared-shredding MAP

[lszskye]

Purpose

Support reading selected keys directly from shared-shredding MAP columns.

Previously, queries such as:

SELECT attrs['key1'], attrs['key2'] FROM T

still read and reconstructed the complete MAP value. This PR connects
__PAIMON_MAP_SELECTED_KEYS with the shared-shredding reader so that only
the required physical columns and overflow data are read.

Changes

• Recognize selected-key MAP read types in FormatReaderMapping.
• Build a pruned physical shared-shredding schema containing only:
  • field mapping;
  • physical columns that may contain the selected keys;
  • overflow column when required.
• Materialize selected keys directly as a logical ROW without rebuilding
  the complete MAP.
• Support missing keys, null MAP values, null values, dynamically mapped
  physical columns, and overflow values.
• Validate that selected-key pushdown:
  • targets a top-level MAP;
  • uses the shared-shredding layout;
  • preserves the complete MAP value type.
• Prevent changing the MAP storage layout of an existing field, because
  files written with different layouts are not compatible.
• Preserve MAP storage-layout and max-columns options when renaming a column.
• Apply Spark selected-key rewriting during physical planning, after Spark
  has created DataSourceV2ScanRelation.
• Add shared Spark E2E coverage for Spark 3.2, 3.3, 3.4, 3.5, 4.0, and 4.1.

Tests

Added coverage for:

• selected physical-column schema pruning;
• reading selected keys from physical columns and overflow;
• missing and null keys;
• Parquet and ORC table reads;
• format-reader mapping and invalid selected-key types;
• MAP storage-layout changes and column renames;
• selected-key pushdown versus full MAP reads;
• normal MAP columns without shared shredding.

[JingsongLi]

Could we preserve the value type from the data schema when constructing the selected-key read field here?

Currently, this branch adds field directly to readDataFields. However, field is the temporary selected-key ROW using the current table schema’s MAP value type. For files written before a MAP value-type evolution, SchemaEvolutionUtil.createIndexCastMapping therefore sees identical expected/read types and does not create the required cast.

I reproduced this with the following sequence:

1. Create a shared-shredding MAP<STRING, INT> column.
2. Write a file containing key1 -> 10.
3. Alter metrics.value from INT to BIGINT.
4. Read metrics['key1'] through selected-key pushdown.

The ORC reader returns 0 instead of 10, while the Parquet reader fails with field 0 is missing.

The selected-key read ROW should use the MAP value type from dataField so that the physical reader reads the old file schema correctly, and the existing schema-evolution cast mapping can then convert that ROW to the current table value type. It would also be good to add an ORC/Parquet regression test covering this schema-evolution case.

[lszskye]

Could we preserve the value type from the data schema when constructing the selected-key read field here?

Currently, this branch adds field directly to readDataFields. However, field is the temporary selected-key ROW using the current table schema’s MAP value type. For files written before a MAP value-type evolution, SchemaEvolutionUtil.createIndexCastMapping therefore sees identical expected/read types and does not create the required cast.

I reproduced this with the following sequence:

1. Create a shared-shredding MAP<STRING, INT> column.
2. Write a file containing key1 -> 10.
3. Alter metrics.value from INT to BIGINT.
4. Read metrics['key1'] through selected-key pushdown.

The ORC reader returns 0 instead of 10, while the Parquet reader fails with field 0 is missing.

The selected-key read ROW should use the MAP value type from dataField so that the physical reader reads the old file schema correctly, and the existing schema-evolution cast mapping can then convert that ROW to the current table value type. It would also be good to add an ORC/Parquet regression test covering this schema-evolution case.

Thank you for pointing this out. You are right—there was indeed a correctness issue in this scenario. I have fixed it and added ORC and Parquet regression tests covering MAP value-type evolution with selected-key pushdown.

[JingsongLi]

• SchemaManager.java:898 only migrated map.storage-layout and map.shared-shredding.max-columns, omitting MAP_SHARED_SHREDDING_COLUMN_PLACEMENT_POLICY (defined in CoreOptions.java:103).
• If the original column is configured as plain/sequential, the old option will remain under the old column name after the rename, and the new column will silently fall back to the default LRU, resulting in changes to subsequent file layout behavior.
• It is recommended to add this option to the rename list and perform additional testing to verify that old keys are deleted and new keys/values are retained.

[lszskye]

• SchemaManager.java:898 only migrated map.storage-layout and map.shared-shredding.max-columns, omitting MAP_SHARED_SHREDDING_COLUMN_PLACEMENT_POLICY (defined in CoreOptions.java:103).
• If the original column is configured as plain/sequential, the old option will remain under the old column name after the rename, and the new column will silently fall back to the default LRU, resulting in changes to subsequent file layout behavior.
• It is recommended to add this option to the rename list and perform additional testing to verify that old keys are deleted and new keys/values are retained.

DONE. I have fixed this issue and added a unit test to verify it.

[JingsongLi]

• When reading old files after a MAP field has been renamed, FormatReaderMapping.Builder.selectedKeysDataField constructs the reader field using expectedField.newType(...), retaining the new logical field name.
• The footer and shredding metadata in old files are still stored using the old physical field names.
• MapSharedShreddingReadPlanFactory.shouldCreateReadPlan looks up metadata based on the reader field name; therefore, selected-key reads of old files cannot create a shredding reconstruction plan, which may result in read failures or incorrect results.
• Regular full MAP reads work because the original schema evolution path retains the physical field names; only the new selected-key path loses them.
• It is recommended to construct the field based on dataField to preserve the old name, then copy the selected-key description/type from expectedField, for example, dataField.newType(...).newDescription(expectedField.description()).
• A regression test for ORC/Parquet is needed to cover the scenario: “write to the old field first → rename → selected-key read of the old file.”