fix for bug when fused pq is used with no hierarchy#664
Merged
Conversation
Contributor
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
|
LGTM |
3 tasks
eolivelli
added a commit
to eolivelli/jvector
that referenced
this pull request
May 9, 2026
* Add on-disk graph index compaction algorithm Introduce OnDiskGraphIndexCompactor and PQRetrainer for streaming N:1 merging of on-disk HNSW indexes without full in-memory materialization. Supports deletion filtering via live-node bitsets, custom ordinal mapping, and PQ codebook retraining. * Add compaction unit tests Tests for OnDiskGraphIndexCompactor covering basic compaction, deletions, ordinal remapping, multi-source merging, and FusedPQ compaction scenarios. * Add reporting and storage infrastructure for CompactorBenchmark Add JFR recording, system stats collection, JSONL logging, git info capture, thread allocation tracking, dataset partitioning, and cloud storage layout utilities used by CompactorBenchmark. Switch jvector-examples logging from logback to log4j2 for consistency with benchmarks-jmh and to avoid duplicate SLF4J bindings in the fat jar. * Add CompactorBenchmark and tooling JMH-based benchmark with configurable workload modes (PARTITION_AND_COMPACT, PARTITION_ONLY, COMPACT_ONLY, BUILD_FROM_SCRATCH), recall measurement, JFR recording, and JSONL result logging. Includes BenchmarkParamCounter for progress tracking, EventLogAnalyzer for post-run analysis, GHA workflow, and exec-maven-plugin integration. Add forced vectorization provider property to VectorizationProvider for benchmark reproducibility. * Update build config and project metadata for compaction Add result file patterns to .gitignore, update rat-excludes for the new compaction workflow and catalog cache files. * Fix JMH jar selection in run-compaction.yml The benchmarks-jmh-*.jar glob matched the -javadoc jar first, which has no Main-Class. Select the shaded JMH jar explicitly by excluding -javadoc and -sources jars. * Fix CompactorBenchmark invocation in run-compaction.yml Use -cp with CompactorBenchmark.main() instead of -jar with JMH Main to avoid BenchmarkList discovery issues in CI's shaded jar. * Address PR review feedback - Extract CompactWriter into its own file to reduce OnDiskGraphIndexCompactor size - Rewrite SystemStatsCollector to read /proc files directly in Java instead of spawning bash - Clarify recall section description in docs/compaction.md * Fix benchmark invocation in docs and default dataset Use -cp instead of -jar in docs since the benchmarks-jmh-*.jar glob matches the -javadoc jar first. Change default dataset from glove-100-angular to ada002-100k. Note -Xmx should be adjusted to fit the dataset. * Fix jar selection: use fixed output name compactor-benchmark.jar The benchmarks-jmh-*.jar glob expands to multiple jars (shaded + javadoc), causing -cp to misinterpret the second jar as the main class. Configure shade plugin outputFile to produce a fixed compactor-benchmark.jar name. Update docs and CI workflow. * Refactor workload modes and fix build-from-scratch timing Simplify WorkloadMode enum: PARTITION_ONLY/COMPACT_ONLY/COMPACT_AND_RECALL/ BUILD_FROM_SCRATCH collapsed into PARTITION/COMPACT/BUILD/PARTITION_AND_COMPACT plus a separate measureRecall flag. Fix buildFromScratch timing to include PQ computation and graph construction (previously only timed the write step). Add fair comparison guidelines to CompactorBenchmark.md. * Add TIERED_10_90 and TIERED_1_99 split distributions Support 10%/90% and 1%/99% partition splits for benchmarking compaction of a small new segment into a large existing index. Add split distribution reference table to CompactorBenchmark.md. * fix for bug when fused pq is used with no hierarchy (datastax#664) --------- Co-authored-by: dian-lun-lin <cyc4542000@gmail.com> Co-authored-by: Mark Wolters <mwolters138@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Bug
CompactWriter.writeFooter() is the counterpart of AbstractGraphIndexWriter.writeSparseLevels(). Both methods write the "source feature" block that sits between the L0 node data and
the actual footer bytes. OnDiskGraphIndex.loadInMemoryFeatures() reads this block to populate hierarchyCachedFeatures, which FusedPQDecoder.similarityTo() uses.
writeSparseLevels() has two branches:
CompactWriter.writeFooter() had only the first branch (via level1FeatureRecords). With enableHierarchy: false, level1FeatureRecords is always empty, so nothing gets written. When
loadInMemoryFeatures then tries to read one entry from what it thinks is that block, it consumes bytes from the footer header instead — populating hierarchyCachedFeatures with one
garbage entry keyed on a non-zero integer. When GraphSearcher.initializeInternal calls scoreFunction.similarityTo(entry.node) (line 331), entry.node is 0 (first remapped ordinal),
which is not in the garbage-keyed map → Node 0 is not in the hierarchy.
The Fix (2 files)
what AbstractGraphIndexWriter writes in its no-hierarchy branch.
completes, reads the entry node's vector from its source graph, encodes it with the retrained PQ, and passes it to the writer before calling writeFooter().