Core: Order Variant metadata dictionary and object fields by UTF-8 byte order - #17726
Open
nssalian wants to merge 1 commit into
Open
Core: Order Variant metadata dictionary and object fields by UTF-8 byte order#17726nssalian wants to merge 1 commit into
nssalian wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Variant metadata stores field names in a dictionary. When its
sorted_stringsflag is set, VariantEncoding.md requires those names to be in UTF-8 byte order.Iceberg sorts and looks them up with Java's
String.compareTo, which is UTF-16 order. For ASCII and most characters the two orders are identical, but they differ for characters above U+FFFF. So Iceberg can flag a dictionary assortedwhen it is not actually UTF-8-ordered. Iceberg reads its own files fine because it sorts and searches the same way, but a spec-conforming reader (iceberg-go, iceberg-rust) searches by UTF-8 and would fail to find those fields.Changes
Sort and look up Variant field names with the existing UTF-8 comparator
Comparators.charSequences()at every site:Variants.metadata- sets the dictionarysorted_stringsflagVariantUtil.find- reader lookup over field namesShreddedObject- shredded object field orderRecordConverter(kafka-connect) andParquetMetrics- sort field names before building the dictionaryNo on-disk format or spec change - this conforms Iceberg to the existing spec. Files with only ASCII or common field names are unaffected; only names above U+FFFF change ordering.
Testing
TestVariantMetadataFieldOrderingand a newTestShreddedObjectcase check that a dictionary is flaggedsortedonly when it is actually UTF-8-ordered, and that fields with characters above U+FFFF are found on read. The tests fail on the oldString.compareTocode and pass on the fix (verified by reverting the change and watching them go red).