Conversation
github-actions
Bot
added
enhancement
mbutrovich
self-requested a review
September 14, 2026 15:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5900. Part of #5818; follows the matched normalization/hash benchmarks in #5822 and singleton optimization in #5887.
Rationale for this change
Normalizing multi-entry
map<string,int>batches slices the key array and allocates Arrow sortingindices and prefix tuples for every row. Reuse the prefix-tuple buffer across rows to reduce this
allocation overhead while preserving the exact permutation.
What changes are included in this PR?
Utf8keys withInt32values, reusing(u32 index, u32 prefix, u64 length)scratch storage. Keep Arrow's comparator and tuple layout, including equal-key permutations.
Scratch capacity grows with the largest row. The existing global index append and single
takeremain.map_sortsingleton normalization (18x faster) #5887's singleton eligibility check, batch dispatch and direct index append. Insert themulti-entry path into its const-generic helper; other key/value types retain Arrow validation.
mixed singleton/multi-entry slices, physical entries under null maps, unsupported keys and empty
visible slices. Retain all upstream tests and the existing null/schema/sorted-flag semantics.
How are these changes tested?
-D warningspassed.make format PROFILES=-Pspark-4.0passed, using the existing profile that provides SemanticDB.slowdown flags with CI lower bound above +1% received independent paired measurements, as did
two wide-map wins. Two fallback cases still above +2% received reverse-order pairs.
Matched 2–10-entry forward normalization improved about 66% (3x); matched 2–50-entry cases improved
28–39% in the full cohort and 33–39% in fresh pairs. No larger slowdown persisted across the
confirmations. The int-key 16-entry fallback was +3.50% in a fresh pair and +1.99% in reverse order;
small effects remain uncertain under observed process drift. The int-key 4-entry case fell from
+6.09% to +0.11% in reverse order. Initial singleton +8–9% flags did not reproduce: fresh matched
singleton pairs were -0.08% and +1.72%. These are microbenchmarks, not end-to-end Spark measurements.
Measurement cohort and limits
Base:
4abfd95114d61ad454f9ee269be1615f469f24e2, including merged singleton PR #5887 (ca223a8d4).All 122 baseline measurements completed before the integrated production source was applied.
Both cohorts used byte-identical benchmark files, dependency manifests and lockfile.
Apple M4 Max, 64 GB, Rust 1.97.1, release/thin LTO/codegen-units=1, no RUSTFLAGS override.
Full cohorts: 50 samples, 1 s warmup, 2 s target measurement (Criterion extends long cases).
Independent pairs: 100 samples, 3 s warmup, 5 s target measurement. No heavy build or benchmark ran
concurrently. Narrow within-process CIs do not capture desktop/background and cross-process drift.
All unfavorable original measurements are retained alongside confirmations.
Inputs and correctness checks are untimed. Normalization output allocation/drop and combined
struct reconstruction/drop are timed. Hash-buffer allocation is untimed and seed reset is timed.
Hash-only forward/reversed cases use the same normalized input as repeated controls. Each batch
has 8,192 visible rows; physical entries under null maps count toward entry rates. No phase subtraction
is used. Hashing implementation and nested-hashing defaults are unchanged.
Complete integrated comparison (122 cases)
Time is ms per 8,192-row batch, with 95% CI; unsupported-key and buffer-reset cases are per call.
The change column is Criterion's cross-process mean estimate; confirmations follow below.
Fresh independent pairs and reverse-order confirmations
Benchmark executable identities (SHA-256)
map_sort:d93a75c7df21c0b57fb994664c6024b8aa07bea85939f08d7b0e6279689c6e8fhash:05739bd6b3ef97c63ed814943cd994e205212abb9a95c8484c977ebdc77e3207map_sort:47a0a73bbedfd8c6339c0e3626d1d0af6ffb9e6fdee679c8c26896905147f7a3hash:ccc68e4d95296c241e222a7c5a552c8a57c30b5365fccbe72ff097a5400e7dd3