feat: support LAION large TopK benchmarks

[jamesgao-jpg]

Related to #826.

Summary

• Parameterize Performance768D100M to select LAION query and ground-truth artifacts for K through 1,000,000.
• Keep wide ground truth path-based and read one Arrow/NumPy row at a time instead of materializing Python integer lists.
• Replace quadratic recall/NDCG work with O(K) scoring and report recall_at across supported depth cutoffs.
• Add serial/concurrent p50 latency and propagate K-aware results through REST and frontend consumers.
• Add first-class response payload profiles to every vector PerformanceCase, with separate IDs-only and vector runs.
• Expose both return scenarios in the frontend for Milvus and Zilliz Cloud, reject unsupported backend/profile combinations before dataset preparation or loading, and preserve payload identity in frontend, REST, and leaderboard results.
• For Milvus and Zilliz Cloud requests above K=16,384, create or validate the collection with query_mode=large_topk before vector index creation and log the automatic mode selection.

Metrics

• Each return scenario reports independent P99 latency, QPS, and recall.
• qps remains the highest successful QPS observed among the configured concurrency levels.
• VDBBench does not attempt to discover a backend concurrency limit.

Compatibility

• Existing LAION runs with K up to 1,000 continue using the original 1,000-query artifacts.
• K from 1,001 through 100,000 uses the hosted 200-query top-100K ground truth.
• K from 100,001 through 1,000,000 uses the hosted 200-query top-1M ground truth.
• Filtered LAION runs above K=1,000 and LAION K above 1,000,000 fail before database initialization.
• IDs-only remains the default payload profile, and legacy custom_case.payload_profile input remains supported.
• Existing result files load with defaults for the new K, latency, recall, and payload fields.
• Self-hosted Milvus requires 2.6.14 or later for query_mode=large_topk; other backends must already permit the requested K.

Verification

• Original implementation: make lint, make unittest, 108 focused dataset, runner, CLI, backend-translation, result-compatibility, REST, and frontend tests, CLI --dry-run validation, and Level 3 impact-map validation.
• Rebase onto main at 5da3ad1: 60 focused large-TopK and affected FTS metric/CLI/result tests passed.
• git diff --check origin/main...HEAD passed after the rebase.
• Exact DCO sign-off verified for every outgoing commit.

Scope Notes

• Milvus and Zilliz Cloud collection setup is automatic for K above 16,384. Reused collections are rejected before load/search when their query mode is incompatible.
• Other backends receive K unchanged and are not reconfigured.
• No concurrency-limit discovery is included.
• The rebase was verified with deterministic tests only; it did not rerun a live Milvus or Zilliz Cloud benchmark probe.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[frankleaf]

#836

Please take a look at this PR to see if it's needed.

[frankleaf]

@jamesgao-jpg
Could you please let me know if this branch is ready for testing now, and if there's any remaining development work?

[jamesgao-jpg]

@jamesgao-jpg Could you please let me know if this branch is ready for testing now, and if there's any remaining development work?

@frankleaf It should be ready for testing, and I did some preliminary testing on Zilliz Cloud already and got some results (not published yet), and the end to end result looks reasonable.

[frankleaf]

@jamesgao-jpg Sure, I'll give it a try as well. By the way, could you let me know when the Zilliz test results will be available? I'd like to take a look and compare the performance differences between Zilliz and open-source Milvus.

[frankleaf]

@jamesgao-jpg Also, I have another question: Is the current 1M GT obtained by performing FLAT brute-force search on the LAION 100M dataset?

[jamesgao-jpg]

@jamesgao-jpg Also, I have another question: Is the current 1M GT obtained by performing FLAT brute-force search on the LAION 100M dataset?

@frankleaf Conceptually yes. In practice we used GPU to parallelize computation and did it streamingly to avoid memory problem, so not exactly flat brute force in CPU as in faiss. I will upload the gt generation script later to the bucket for reproductibility.

[frankleaf]

@jamesgao-jpg I've tested it, and the 1M recall is working perfectly—great job! Also, I was wondering if there are any plans to provide a 1M test set with filters along with ground truth?

[jamesgao-jpg]

@jamesgao-jpg I've tested it, and the 1M recall is working perfectly—great job! Also, I was wondering if there are any plans to provide a 1M test set with filters along with ground truth?

@frankleaf Yes that will be the plan next. I will pin you once that's released.

[frankleaf]

@jamesgao-jpg thanks! Also, will we be launching a separate leaderboard for the Top1M scenario on https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch ?

[jamesgao-jpg]

@jamesgao-jpg thanks! Also, will we be launching a separate leaderboard for the Top1M scenario on https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch ?

@frankleaf Yes that will be the plan, but might not be released as soon, since we need to decide what's the best leaderboard format to introduce this and what backends can provide meaningful results (many backends support very limited topks right out of the box).