feat(core, indexer): support group_by searching and fix hnsw sparse by JalinWang · Pull Request #527 · alibaba/zvec

JalinWang · 2026-06-25T08:07:31Z

Summary

This PR implements group_by search (a.k.a. group-by deduplication) across the core interface, algorithm, and DB layers of zvec. Users can now retrieve top-K results per group instead of globally, with support for fetch_vector, is_linear, and bf_pks query modes on supported index types. group_by is mutually exclusive with refiner search and is rejected when refiner_param is set.

Motivation

Prior to this PR, group_by was either silently ignored (IVF, Vamana) or produced incorrect/empty results (DiskAnn, sparse fetch_vector). This PR brings consistent, well-tested group_by behavior to all index types.

Current Status

Unsupported index types now fail fast with IndexError_Unsupported and an error log, instead of silently returning empty or incorrect results.

Support	Reject
flat (dense)	diskann
flat (sparse)	ivf
hnsw (dense)	vamana
hnsw (sparse)
hnsw_rabitq

GroupBy search mode compatibility:

Search mode with `group_by`	Status
`fetch_vector` `is_linear` `bf_pks`	✅
refiner search (`refiner_param`)	Rejected

We will gradually enable GroupBy searching for them.

Previous Status

Index Type	group_by (graph search)	group_by + is_linear	group_by + bf_pks	fetch_vector in group_by	Notes
flat (dense)	N/A (always linear)	Yes	Yes	Yes	Full support via FlatStreamerContext
flat_sparse	N/A (always linear)	Yes	Yes	No	Known limitation; search_group/search_group_p_keys don't emit vectors
hnsw (dense)	Yes	Yes	Yes	Yes	Graph search via HnswAlgorithm::search, bf via HnswStreamer::search_bf_impl and search_bf_by_p_keys_impl
hnsw_sparse	Yes	Yes	Yes	Partial (no sparse vector in group result)	topk_to_group_result uses get_vector_meta(id) only, not full sparse data
hnsw_rabitq	Yes	Yes	Yes	Yes	Both graph and bf paths handle group_by; HnswRabitqContext has full group result + fetch
diskann	Yes (buggy)	Yes (buggy)	Yes (buggy)	No	Empty results — linear_search/keys_search never populated group_topk_heaps_; knn_search had typo at L985 writing to wrong heap; and the fetch vector doesn’t work in normal search
ivf	No	No	No	No	IVFSearcherContext has no group_by_search(), no group_topk_heaps. group_by is silently ignored
vamana	No	No	No	No	VamanaContext has no group support. topk_to_result only builds flat results. group_by is silently ignored

Key Changes

Core Interface Layer

index_param.h: Added GroupByParam struct (group_topk, group_count, group_by callback) and group_by_param field on BaseIndexQueryParam.
index.h / index.cc: Added group_doc_list_ to SearchResult. Introduced for_each_doc helper to uniformly iterate over flat and grouped results for score normalization and vector reverting. Added _set_group_by_on_context static helper called by supported index types at the end of _prepare_for_search. Added a fast rejection for group_by combined with refiner search.
Per-index _prepare_for_search: FlatIndex, HNSWIndex, HNSWRabitqIndex call _set_group_by_on_context. IVFIndex, DiskAnnIndex, and VamanaIndex add early rejection checks for group_by.

Algorithm Layer

flat_sparse_search.h: Populates IndexSparseDocument in group_by results when fetch_vector is enabled (previously missing).
hnsw_sparse_context.h: Fetches full sparse vector data via get_sparse_data + SparseUtility::ReverseSparseFormat in group results (previously only stored get_vector_meta).

DB Layer

engine_helper.hpp: Translates DB-layer vector_column_params::GroupByParams to core-layer GroupByParam when building the engine query param.
vector_column_indexer.cc: Returns GroupVectorIndexResults when search_result.group_doc_list_ is populated.
combined_vector_column_indexer.cc: Merges group_by results across multiple index blocks — adjusts doc keys by block offset, merges groups by group_id, sorts within groups by score (respecting metric direction), and truncates to group_topk / group_count.

Tests

index_group_by_test.cc (new, 520 lines): Data-driven GroupByInterfaceTest fixture at the core interface layer with RunOk/RunRejected methods. Covers dense (flat, hnsw, hnsw_rabitq — graph, linear, bf_pks, fetch_vector), sparse (flat, hnsw — graph, linear, bf_pks, fetch_vector), and unsupported (vamana, ivf, diskann) index types. Also verifies that group_by and refiner search are mutually exclusive.
vector_column_indexer_test.cc (+257 lines): Data-driven GroupByIndexerTest fixture at the DB indexer layer, mirroring the core test pattern. Covers dense (flat, hnsw — graph, linear, bf_pks, fetch_vector), sparse (flat, hnsw), and unsupported (ivf, diskann with optional plugin skip).

…ssing Added for_each_doc helper to unify iteration over flat doc_list_ or grouped group_doc_list_, eliminating 5 of 6 duplicated if(has_group_by) blocks in _dense_search and _sparse_search. Reduces code duplication and makes the group_by logic easier to maintain. The one remaining dual-rail (reformer normalize) is preserved because the batch API requires different handling for grouped vs flat results.

Move DiskAnn group_by support to feat/group_by_diskann branch. On this branch, DiskAnn explicitly rejects group_by with IndexError_Unsupported, matching Vamana and IVF behavior.

iaojnh · 2026-06-25T11:28:19Z

+    uint32_t group_count =
+        query_params.group_by ? query_params.group_by->group_count : 0;
+    if (group_count > 0 && merged_group_docs.size() > group_count) {
+      merged_group_docs.resize(group_count);


这里截断的依据是什么，好像没看到merged_group_docs有排过序？

确实掉了，已加上；后续单独pr重构此函数

iaojnh · 2026-06-25T11:31:18Z

+      (search_param->group_by_param && search_param->group_by_param->group_by);
+  if (has_group_by) {
+    result->group_doc_list_ = std::move(
+        const_cast<core::IndexGroupDocumentList &>(context->group_result()));


minor：可以考虑给context弄一个mutable_group_result方法，这样就不需要const_cast

iaojnh · 2026-06-25T11:35:37Z

+  // Return grouped results when group_by is active
+  if (!search_result.group_doc_list_.empty()) {
+    auto result = std::make_shared<GroupVectorIndexResults>(
+        std::move(search_result.group_doc_list_));


reverted_vector_list_ 和 reverted_sparse_values_list_ 应该还是需要的吧

iaojnh · 2026-06-25T11:37:43Z

      std::min(256u, vamana_search_param->prefetch_lines);
  params.set(core::PARAM_VAMANA_STREAMER_PL, real_search_pl);
  context->update(params);
+  _set_group_by_on_context(search_param, context);


vamana不支持groupby，这里还需要set context吗？

iaojnh · 2026-06-25T11:44:59Z

      return core::IndexError_Runtime;
    }

    auto &base_result = context->result();


groupby search的情况下，这个result应该是空的

确实，refiner和group by我先互斥掉吧

egolearner · 2026-07-01T09:14:27Z

+          doc.set_key(block_offsets_[i] + doc.key());
+        }
+        // Merge into existing group or create new one
+        auto it = group_merge_map.find(group.group_id());


可以使用operator []，不存在自动创建

重构后不需要区分no exist状态了，done

…tor' into feat/group_by

iaojnh · 2026-07-03T08:42:19Z

+    if (has_group_by) {
+      for (auto &group : result->group_doc_list_) {
+        auto *docs = group.mutable_docs();
+        core::IndexDocumentList doc_list(docs->begin(), docs->end());


这个doc_list需要吗，normalize直接传docs是不是就好了

疏忽了，确实，done

iaojnh · 2026-07-03T09:16:27Z

+      merged_docs.reserve(merged_docs.size() + docs.size());
+
+      for (size_t doc_idx = 0; doc_idx < docs.size(); ++doc_idx) {
+        auto doc = docs[doc_idx];


这里为何不和VectorResultAccumulator::AddBlock的实现一样，使用std::move

JalinWang added 15 commits June 18, 2026 16:12

feat: init

989b38c

fix: sparse fetch in groupby; diskann fetch; check compatibility; fix ut

c512798

fix: ut

0239c18

fix: diskann groupby fails loudly

daf85ef

fix: indexer test link diskann

1a706d1

diskann support groupby

44dfc0a

refactor test

7aa16ca

use merge to build

49f5a31

fix: diskann fetch

aa0acfe

refactor: diskann context restoration

f49e0ad

revert: diskann group_by support; reject group_by for DiskAnn

527945b

Move DiskAnn group_by support to feat/group_by_diskann branch. On this branch, DiskAnn explicitly rejects group_by with IndexError_Unsupported, matching Vamana and IVF behavior.

revert: diskann related

d15896e

revert: headers

fb7446b

test: refactor ut and add flat sparse

d47e1e7

JalinWang changed the title ~~Feat/group by~~ feat(core, indexer): support group_by searching Jun 25, 2026

Merge branch 'main' into feat/group_by

d5c1a10

JalinWang marked this pull request as ready for review June 25, 2026 08:08

JalinWang requested review from chinaux, iaojnh, richyreachy and zhourrr as code owners June 25, 2026 08:08

JalinWang removed the request for review from chinaux June 25, 2026 08:23

JalinWang changed the title ~~feat(core, indexer): support group_by searching~~ feat(core, indexer): support group_by searching and fix hsnw sparse Jun 25, 2026

JalinWang requested review from egolearner and feihongxu0824 and removed request for zhourrr June 25, 2026 08:24

github-actions Bot assigned JalinWang Jun 25, 2026

Merge branch 'main' into feat/group_by

17d3f8a

iaojnh reviewed Jun 25, 2026

View reviewed changes

JalinWang changed the title ~~feat(core, indexer): support group_by searching and fix hsnw sparse~~ feat(core, indexer): support group_by searching and fix hnsw sparse Jun 26, 2026

JalinWang added 13 commits June 30, 2026 15:31

Reject group_by refiner search

91d9c76

rm vamana _set_group_by_on_context

846528c

Fix grouped vector index results

a563c1a

add ut for combined indexer & move to a seperate file

a9babe0

Add mutable group result accessor

c13781c

chore: reformat

9beec0f

Merge branch 'main' into feat/group_by

0bbb381

Refactor combined vector search result merging

9a56cab

comment

a5308f3

fix: diskann visit filter duplicate definition

dce2f3a

fix: early reject group by before train()

603b906

revert reformt on un-related files

52e7ae7

revert

fb1622a

egolearner reviewed Jul 1, 2026

View reviewed changes

Comment thread src/db/index/column/vector_column/combined_vector_column_indexer.cc Outdated

egolearner reviewed Jul 1, 2026

View reviewed changes

Comment thread src/db/index/column/vector_column/combined_vector_column_indexer.cc Outdated

egolearner reviewed Jul 1, 2026

View reviewed changes

Comment thread src/include/zvec/core/interface/index.h

JalinWang added 8 commits July 2, 2026 10:39

fix: crash during ut cleaning after early rejected

dcda63e

fix: split for_each_doc function

0d3260a

fix: use group_reverted_vector_list_

900600c

fix: MV is_group_by_unsupported_index

dfaf964

Merge remote-tracking branch 'fork/codex/combined-search-merger-refac…

b33d422

…tor' into feat/group_by

fix: use sort instead of ResultDoc

81a6fb2

fix: use [] to insert

905d04d

Merge branch 'main' into feat/group_by

2f8369e

iaojnh reviewed Jul 3, 2026

View reviewed changes

JalinWang added 2 commits July 6, 2026 10:44

reduce copy

d4afc36

Merge branch 'main' into feat/group_by

9083ff0

Uh oh!

Conversation

JalinWang commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Current Status

Previous Status

Key Changes

Core Interface Layer

Algorithm Layer

DB Layer

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JalinWang commented Jun 25, 2026 •

edited

Loading