Skip to content

Optimize populated run-optimized BSI query results - #539

Open
perfloop-agent wants to merge 3 commits into
RoaringBitmap:masterfrom
perfloop:perfloop-pr-open-66gs2g3x13
Open

Optimize populated run-optimized BSI query results#539
perfloop-agent wants to merge 3 commits into
RoaringBitmap:masterfrom
perfloop:perfloop-pr-open-66gs2g3x13

Conversation

@perfloop-agent

@perfloop-agent perfloop-agent commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

The parallel BSI query paths (BatchEqual, CompareValue) called RunOptimize on every worker-local result bitmap while those bitmaps were still empty, paying per-worker optimization work that can never compress anything. This PR moves run optimization to the populated result:

  • Worker-local result bitmaps are no longer run-optimized while empty; the aggregate returned by the parallel executor is optimized once, after all matches have been merged. BSI aggregate results get the same treatment.
  • Empty results remain mutable, so callers can keep writing into them.
  • The setter growth path no longer attempts to optimize an empty slice using an invalid index.

Populated contiguous results keep run compression exactly as before; only the wasted empty-bitmap passes are gone.

Performance

Measured with adjacent before/after builds, fresh benchmark processes, and isolated build caches:

workload metric before after
CompareValue result serialization, 50k contiguous matches, 4 workers ns/op ~2600 ~187
same B/op 9600 136
same allocs/op 4 4

CompareValue end-to-end latency on the same workload was unchanged within noise, and BatchEqual moved ~30 ns/op on a ~1.7 ms operation — no material regression on any probed shape (alternating, sparse-container, single-match, and scattered variants).

Testing

  • go test ./... and go test -race ./BitSliceIndexing pass.
  • A differential probe covered signed values through the 64-bit boundaries, invalid and duplicate queries, the large scattered-query path, empty-result mutability, aggregate compression, and setter growth; all outputs and state checks match.

The branch includes a merge of current master; the only overlap was the shared BSI benchmark/test file, resolved by keeping the superset that contains both the existing BatchEqual coverage and the new run-optimized benchmarks.

Full verification record: https://app.perfloop.ai/t/oss/case_q4eynnf09a

@perfloop-agent
perfloop-agent force-pushed the perfloop-pr-open-66gs2g3x13 branch from da7ff3d to aa3338d Compare August 4, 2026 02:24
@perfloop-agent perfloop-agent changed the title Optimize populated BSI query results Optimize populated run-optimized BSI query results Aug 4, 2026
@tsenart
tsenart force-pushed the perfloop-pr-open-66gs2g3x13 branch from 5268cd3 to 23d4df9 Compare August 4, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant