Skip to content

Core: Order Variant metadata dictionary and object fields by UTF-8 byte order - #17726

Open
nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:variant-metadata-utf8-field-ordering
Open

Core: Order Variant metadata dictionary and object fields by UTF-8 byte order#17726
nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:variant-metadata-utf8-field-ordering

Conversation

@nssalian

Copy link
Copy Markdown
Collaborator

Rationale for this change

Variant metadata stores field names in a dictionary. When its sorted_strings flag is set, VariantEncoding.md requires those names to be in UTF-8 byte order.

Iceberg sorts and looks them up with Java's String.compareTo, which is UTF-16 order. For ASCII and most characters the two orders are identical, but they differ for characters above U+FFFF. So Iceberg can flag a dictionary as sorted when it is not actually UTF-8-ordered. Iceberg reads its own files fine because it sorts and searches the same way, but a spec-conforming reader (iceberg-go, iceberg-rust) searches by UTF-8 and would fail to find those fields.

Changes

Sort and look up Variant field names with the existing UTF-8 comparator Comparators.charSequences() at every site:

  • Variants.metadata - sets the dictionary sorted_strings flag
  • VariantUtil.find - reader lookup over field names
  • ShreddedObject - shredded object field order
  • RecordConverter (kafka-connect) and ParquetMetrics - sort field names before building the dictionary

No on-disk format or spec change - this conforms Iceberg to the existing spec. Files with only ASCII or common field names are unaffected; only names above U+FFFF change ordering.

Testing

TestVariantMetadataFieldOrdering and a new TestShreddedObject case check that a dictionary is flagged sorted only when it is actually UTF-8-ordered, and that fields with characters above U+FFFF are found on read. The tests fail on the old String.compareTo code and pass on the fix (verified by reverting the change and watching them go red).

@nssalian
nssalian marked this pull request as ready for review August 19, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant