Skip to content

perf(parquet): encode dictionary indices in RLE batches - #1328

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-batch-dict-indices
Open

fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-batch-dict-indices

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

What

  • Encode Parquet dictionary indices in RLE batches.

Why

  • WriteIndices currently sends every index through Put, including long repeated runs and complete literal groups.

Implementation

  • Add PutBatchIndices for int32 indices.
  • Encode repeated runs and complete literal groups directly.
  • Pack literal groups through one batched bit-writer call.
  • Keep scalar handling for partial literal groups.
  • Preserve encoded bytes and dictionary state on write errors.

Benchmark

Command, run six times per version with before/after order alternated:

go test -run '^$' -bench '^BenchmarkDictEncoderWriteIndices$' -benchmem -benchtime=500ms -count=1 ./parquet/internal/encoding

Apple M1 Pro, Go 1.26.3, GOMAXPROCS=1. WriteIndices on 65,536 indices.

Input Before After Time change
Constant 187.7 µs 26.0 µs -86.1%
Runs of 8 388.9 µs 200.9 µs -48.3%
Runs of 32 219.4 µs 67.0 µs -69.5%
Runs of 256 197.6 µs 41.9 µs -78.8%
Random, 16 entries 580.7 µs 306.3 µs -47.3%
Random, 256 entries 452.8 µs 302.8 µs -33.1%
Alternating 400.7 µs 270.6 µs -32.5%

Allocations stay at 232 B/op and 4 allocs/op. The benchmark also covers 64-index batches.

Compatibility

  • Encoded bytes match the existing scalar path.
  • No Parquet format or public API changes.

Tests

  • go test ./parquet/...
  • go test -race ./parquet/internal/utils ./parquet/internal/encoding
  • go test -tags noasm ./parquet/internal/utils ./parquet/internal/encoding
  • go vet -composites=false ./parquet/internal/utils ./parquet/internal/encoding

@samueleresca

Copy link
Copy Markdown
Member

Reproduced on linux/amd64 (AMD Ryzen 9 9950X, Go 1.26.0): BenchmarkDictEncoderWriteIndices is 33–86% faster across all 14 cases (geomean −55%), with no change in allocations (232 B/op, 4 allocs/op). six runs per side, alternating order, benchstat p=0.002 on every case.

Small nit: PutBatchLevels int16 and int32 functions can be merged into a generic. Not a blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants