Skip to content

Reuse dictionary value hashes in partial hash aggregation #26812

Open

Description

@Rich-T-kid

Related to #26780.

PartialAggregate currently hashes dictionary values in two places:

input DictionaryArray
        |
        +-- MultiGroupBy::intern -> create_hashes(values)
        |
        `-- DictionaryGroupValuesColumn -> hashes values for its lookup

Both paths use the aggregation hash seed. The dictionary keys may change between batches while the values Arc remains the same, so the same values are hashed again even though only the O(rows) key scatter is new work.

This is especially visible for grouped dictionary columns produced directly by Parquet scans. Low cardinality keeps the waste small, while medium or high cardinality makes it significant across many input batches.

The partial aggregate should compute dictionary value hashes once per values-array identity and hash state, then reuse them for both group-ID calculation and DictionaryGroupValuesColumn lookup. Composite group keys still need their normal per-row hash combination.

Completion criteria:

  • one values array is hashed once per compatible aggregation hash state
  • both group-ID calculation and dictionary value lookup reuse those hashes
  • the cache is bounded and included in aggregation memory accounting
  • tests cover multiple batches sharing values, a changed values Arc, null keys/values, and composite group keys
  • instrumentation verifies values hashed rather than inferring reuse only from elapsed time

Activity

  1. Rich-T-kid commented on Oct 27, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions