Skip to content

perf(parquet): reuse DELTA decoder scratch - #1322

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decoder-scratch
Open

fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decoder-scratch

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Reuse DELTA length and prefix scratch buffers across decoder pages.
  • Add tests for scratch reuse and page-to-page correctness.

Why

  • The column reader caches decoders by encoding.
  • Each page currently allocates fresh length and prefix arrays even when the decoder already has enough capacity.

Implementation

  • Keep stable backing buffers for decoded lengths and prefix lengths.
  • Expose sliced views to the existing Decode and Discard paths.
  • Keep the existing page-size behavior when a larger page needs more capacity.
  • The existing repeated SetData + Decode benchmark now shows 18 to 15 allocations/op and about 512 KiB less allocation for a full 65,536-value prefix-heavy page.

Tests:

  • go test ./parquet/internal/encoding -count=1
  • go test ./parquet/file -run '^(TestWithEOFReader|TestInvalidHeaders|TestInvalidFooter|TestIncompleteMetadata|TestDeltaLengthByteArrayPackingWithNulls|TestDeltaBinaryPackedMultipleBatches|TestPageStreaming.*|TestPrimitiveReader|TestFullSeekRow|TestSkipEmptyRepeatedRows)$' -count=1
  • go test -race ./parquet/internal/encoding -run 'TestDelta(ByteArrayDecoderReusesPageScratch|LengthByteArrayDecoderReusesLengthScratch|ByteArrayDecoderKeepsResultsAcrossPages|ByteArrayDecoderRejectsInvalidPrefixes)$' -count=1
  • go vet ./parquet/internal/encoding

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant