You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reuse dictionary value hashes in partial hash aggregation #26812
PartialAggregate currently hashes dictionary values in two places:
input DictionaryArray
|
+-- MultiGroupBy::intern -> create_hashes(values)
|
`-- DictionaryGroupValuesColumn -> hashes values for its lookup
Both paths use the aggregation hash seed. The dictionary keys may change between batches while the values Arc remains the same, so the same values are hashed again even though only the O(rows) key scatter is new work.
This is especially visible for grouped dictionary columns produced directly by Parquet scans. Low cardinality keeps the waste small, while medium or high cardinality makes it significant across many input batches.
The partial aggregate should compute dictionary value hashes once per values-array identity and hash state, then reuse them for both group-ID calculation and DictionaryGroupValuesColumn lookup. Composite group keys still need their normal per-row hash combination.
Completion criteria:
one values array is hashed once per compatible aggregation hash state
both group-ID calculation and dictionary value lookup reuse those hashes
the cache is bounded and included in aggregation memory accounting
tests cover multiple batches sharing values, a changed values Arc, null keys/values, and composite group keys
instrumentation verifies values hashed rather than inferring reuse only from elapsed time
Activity
Rich-T-kid commentedon Oct 27, 2026
take