Skip to content

feat(parquet): apply name mapping to files without field IDs - #1044

Open
fairytale0828 wants to merge 1 commit into
apache:mainfrom
fairytale0828:feat/parquet-name-mapping
Open

fairytale0828 wants to merge 1 commit into
apache:mainfrom
fairytale0828:feat/parquet-name-mapping

Conversation

@fairytale0828

Copy link
Copy Markdown
Contributor

Summary

Parquet files without field IDs (for example, files added by table migration) currently fail with NotImplemented in BuildProjection. This PR resolves the TODO by applying the name mapping from ReaderOptions::name_mapping, which the Avro reader already supports.

  • If the Parquet schema has no field IDs, ApplyNameMapping walks the SchemaManifest and adds PARQUET:field_id metadata to each field found in the mapping. The existing projection then runs unchanged.
  • Names follow the Iceberg name mapping spec: list elements use element, and map entries use key and value, regardless of the physical names in the file. Both list and large_list are handled.
  • Fields not in the mapping, or mapped without a field ID, get no ID and follow the existing missing-column rules (initial default, null, or an error for required fields). This matches the Java Parquet reader.
  • Field IDs in the file take precedence. The mapping is applied only when the file has no field IDs at all.
  • A file without field IDs and without a name mapping now returns InvalidSchema, consistent with Avro.

Known limitations, not changed in this PR:

  • NameMapping builds its lookup indexes lazily on first use, so concurrent readers sharing one mapping can race on that first lookup. This already affects Avro and can be fixed separately.
  • Field names containing . cannot be told apart from nested paths, because NameMapping::Find joins path segments with ..

Test Plan

  • ParquetNameMappingTest in parquet_schema_test.cc: assigning IDs by column name, matching an alias, nested struct/list/map fields with non-standard physical names (item, key_value), and unmapped fields or fields mapped without an ID.
  • New cases in ParquetReaderTest in parquet_test.cc:
    • ReadWithoutFieldIdsOrNameMapping: returns InvalidSchema.
    • ReadWithNameMapping: alias, column reordering, type promotion, and a missing optional column.
    • ReadWithFieldIdsAndNameMapping: file field IDs win over a conflicting mapping.
    • ReadNestedFieldsWithNameMapping: end-to-end read of struct, list<struct>, and map<string, struct>, including null and empty values, with both list and large_list.
  • pre-commit run passes on the changed files.
  • ctest -R parquet_test: 140 passed, 1 skipped.

@fairytale0828
fairytale0828 force-pushed the feat/parquet-name-mapping branch from 814bbf4 to c794884 Compare October 20, 2026 02:15

if (!HasFieldIds(metadata->schema()->schema_root())) {
if (name_mapping == nullptr) {
return InvalidSchema(

@shangxinli shangxinli Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java does not error here - ReadConf.java:85-88 falls back to addFallbackIds and assigns positional ids, so this file still reads there. Not a regression, but is closing that gap out of scope?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants