Skip to content

perf(arrow/array): compare optional validity bitmaps in bulk - #1327

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/arrow-validity-bitmap-equality
Open

fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/arrow-validity-bitmap-equality

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

What

  • Speed up Equal and ApproxEqual when one array has no validity bitmap.

Why

  • Comparing an absent bitmap with a materialized bitmap currently checks validity one row at a time.

Implementation

  • Use BitmapAllSet on the present bitmap, scanning aligned words and respecting its slice offset.
  • Return as soon as an invalid bit is found instead of counting the full range.
  • Preserve empty-buffer handling and bitmap checks when declared null counts are inconsistent.

Benchmark

Command: go test -run '^$' -bench '^BenchmarkArrayEqualValidityBitmap$' -benchmem ./arrow/array

Apple M1 Pro, Go 1.26.3, GOMAXPROCS=1. Medians of six alternating before/after runs, 300 ms per case. Each case compares 1,048,576 values with one absent bitmap and one materialized bitmap. Binary/string values are 8 bytes each.

Case Before After Speedup
Boolean, Equal 8,806.5 µs 21.3 µs 414.4x
Boolean, ApproxEqual 4,643.2 µs 20.9 µs 221.9x
Int8, Equal 5,868.4 µs 58.9 µs 99.6x
Int64, Equal 7,586.4 µs 494.7 µs 15.3x
Binary, Equal 11,979.7 µs 1,658.2 µs 7.2x
String, Equal 8,959.1 µs 1,895.6 µs 4.7x
String, ApproxEqual 19,381.8 µs 11,087.5 µs 1.7x

No new allocations. The benchmark also covers 64-value arrays, unaligned slices, and controls with both bitmaps present or both absent.

Compatibility

  • No data-format changes.
  • Adds one reusable bitmap utility; null semantics and slice offsets remain covered by the regression tests.

Tests

  • go test ./arrow/...
  • go test -race ./arrow/array
  • go test -tags noasm ./arrow/array
  • go vet -composites=false ./arrow/array

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking malformed-input edge case: BitmapAllSet([]byte{0xff}, 64, 0) panics. With n == 0, both loops are skipped, but the word-scan still evaluates buf[8:8], which is out of bounds for a one-byte slice. This also changes equality for a zero-length hand-built ArrayData with an out-of-buffer offset from vacuously true to panic. Normal validated arrays cannot reach it. An early if n == 0 { return true } preserves the expected empty-range contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants