Cuvs lucene move by imotov · Pull Request #2353 · NVIDIA/cuvs

imotov · 2026-07-22T22:39:19Z

Moves cuvs-lucene into java/cuvs-lucene folder of cuVs.

This is a preliminary PR to validate CI scripts. We'll need to rerun parts of the process and rebase the PR when we're ready to make the move. So, please ignore all changes in java/cuvs-lucene directory except for build-related changes.

Open questions:

Do we want to have a separate cuvs-lucene-codeowners or do we use cuvs-java-codeowners?

Timeline for the move:

Right after 26.08 release
- Merge this PR to main with no-squash and move all new cuvs-lucene development to cuvs wi
- Add a note to cuvs-lucene that all new development and all issues should be now go to this repo
Between 26.08 release and 26.10 code freeze
- Move all still relevant cuvs-lucene PRs and issue to cuvs repo, close the rest.
- Update release process to publish cuvs-lucene jars from this project instead of cuvs-lucene
- Remove references to the cuvs-lucene repo in various other parts of our infrastructure
  - rapids-metadata
  - workflows (nightly builds)
After 26.10 code freeze and when all issue and PRs are move to this repo
- Archive cuvs-lucene

Preparation script for cuvs-lucene:

git clone git@github.com:NVIDIA/cuvs-lucene.git cuvs-lucene-move
cd cuvs-lucene-move
git filter-repo --to-subdirectory-filter java/cuvs-lucene --replace-message ../message-filter.txt
cd ../cuvs
git remote add cuvs-lucene-origin ../cuvs-lucene-move
git fetch cuvs-lucene-origin
git merge --allow-unrelated-histories cuvs-lucene-origin/main
# Comment message:
# Merge main branch of 'cuvs-lucene' into cuvs
git remote remove cuvs-lucene-origin

message-filter.txt

regex:(?<![\w/])#(\d{1,3})\b==>NVIDIA/cuvs-lucene#\1

See NVIDIA/cuvs-lucene#168

…s-lucene#2) Fixes NVIDIA/cuvs-lucene#1 Things included in this PR: - Add the spotless maven plugin configuration. - Add the gitignore file. - Add a source license-header file that is used by the spotless plugin (to add/update license headers automatically if missing). - Update the readme file. Authors: - Vivek Narang (https://github.com/narangvivek10) - Gil Forsyth (https://github.com/gforsyth) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - MithunR (https://github.com/mythrocks) URL: NVIDIA/cuvs-lucene#2

Fixes NVIDIA/cuvs-lucene#8 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#9

Fixes NVIDIA/cuvs-lucene#5 Main changes in this PR: - Adapt to the cuVS-Java API updates: Mainly replace the use of `Dataset` with the new `CuVSMatrix` - Disable the tests that use byte vector encoding, as we do not support it. Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#10

Add CI files and workflows

This PR adds a few additional needed changes to enable CI. - change permission to make `.sh` files executable - removes unnecessary configuration from `.pre-commit-config.yaml`, `CODEOWNERS`, `dependencies.yaml` - updates all `pre-commit` hooks - adds `rapids-dependency-file-generator` hook (to validated `dependencies.yaml`) - fixes all issues found by `pre-commit` hooks (broken shell scripts, trailing whitespace, etc.) - fixes versioning in `pom.xml` Authors: - Dante Gama Dessavre (https://github.com/dantegd) - James Lamb (https://github.com/jameslamb) Approvers: - James Lamb (https://github.com/jameslamb) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#15

* updates `rapids-dependency-file-generator`, to pull in changes from rapidsai/dependency-file-generator#163 * removes unnecessary error-trapping in shell scripts Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: NVIDIA/cuvs-lucene#19

Contributes to rapidsai/build-planning#208 * uses CUDA 13.0.0 to build and test (using the same patterns from the `cuvs-java` tests, in NVIDIA#1273) ## Notes for Reviewers This switches GitHub Actions workflows to the `cuda13.0` branch from here: rapidsai/shared-workflows#413 A future round of PRs will revert that back to `branch-25.10`, once all of RAPIDS supports CUDA 13. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Ben Frederickson (https://github.com/benfred) - rhdong (https://github.com/rhdong) URL: NVIDIA/cuvs-lucene#20

Contributes to rapidsai/build-planning#208 Now that rapidsai/shared-workflows#413 is merged, this converts all GitHub Actions references from `@cuda13.0` back to `branch-25.10`. ## Notes for Reviewers This is safe to admin-merge because the change is a no-op... configs on those 2 branches are identical.

…cene#22) This PR configures this repo (adds a `.github/release.yml` file) for automatic release notes generation. See https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes. This will fix such issues as hotfix releases including wrong content (e.g [cugraph v25.04.01](https://github.com/rapidsai/cugraph/releases/tag/v25.04.01) contains 50+ PRs from the full v25.04.00 release when it should only contain the single hotfix PR (#5017).) Example auto-generated release notes: https://github.com/rapidsai/literate-octo-potato/releases/tag/v25.10.00a

… prefiltering (NVIDIA/cuvs-lucene#14) Refactoring, CI fixes (pulling libcuvs from pypi if not found), prefiltering support. Added tests: * TestCuVSGaps (for missing vectors in documents) * TestCuVSDeletedDocuments (for deleted vectors, that will leverage prefiltering) * TestMerge (dedicated test for testing merges) Authors: - Ishan Chattopadhyaya (https://github.com/chatman) - Vivek Narang (https://github.com/narangvivek10) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#14

Introducing a new Codec that uses CAGRA for building the index on GPU and serializing to Lucene-compatible HNSW index segments. The Lucene-compatible segments are searchable via the `Lucene99HnswVectorsReader` (which is the default in Lucene 10.x). Note: This is based on top of NVIDIA/cuvs-lucene#14 and should be rebased once that is merged. TODO: - Benchmarks and more tests - Further refactoring to split the `CuVSVectorsFormat` into GPU and CPU-specific formats. Fixes NVIDIA/cuvs-lucene#13 Authors: - Vivek Narang (https://github.com/narangvivek10) - Puneet Ahuja (https://github.com/punAhuja) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - Corey J. Nolet (https://github.com/cjnolet) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: NVIDIA/cuvs-lucene#16

Fixes NVIDIA/cuvs-lucene#7 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#24

…n to give detailed stack traces and information on tests (NVIDIA/cuvs-lucene#25) Fixes NVIDIA/cuvs-lucene#26 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#25

…cene upgrades (NVIDIA/cuvs-lucene#29) As per [last night's comment on the Solr Jira issue](https://issues.apache.org/jira/browse/SOLR-17892?focusedCommentId=18024203&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18024203) we realized that exposing the Lucene Codecs in the SPI registry prevents any calling application from being able to upgrade the codec because, under the hood, the next version of Lucene moves the older codec (Lucene101) into a different package name. As a result, we should only provide the relevant vector format, reader, and writer, and the calling application needs to wrap them into its own FilterCodec. For the sake of the current tests, we have decided to move the codec classes into the tests so that they are not bundled in the artifacts. This issue is a blocker for the inclusion of cuvs Lucene in Solr or any other Lucene-based search engine. Authors: - Vivek Narang (https://github.com/narangvivek10) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - MithunR (https://github.com/mythrocks) URL: NVIDIA/cuvs-lucene#29

…cuvs-lucene#30) We were inadvertently copying over the vectors from a list to an array before creating the CuVSMatrix instance. This is memory inefficient. Authors: - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#30

…VIDIA/cuvs-lucene#31) * cuvs-java 25.10 is released, use that from Maven Central * There's a blocker bug for Solr integration where single vector indexes cause a Solr crash. Adding a graceful handling for that. * Updated README.md with a more accurate description of the project. Authors: - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#31

## Description Supports rollout of new branching strategy. https://docs.rapids.ai/notices/rsn0047/ xref: rapidsai/build-planning#224

Contributes to rapidsai/build-planning#224 ## Notes for Reviewers This is safe to admin-merge because the change is a no-op... configs on those 2 branches are identical.

…vs-lucene#28) Fixes NVIDIA/cuvs-lucene#27 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#28

This commit fixes the publication of cuvs-lucene artifacts to Github. The Github actions build-logs indicate that while the JAR is built correctly, it doesn't seem to be uploaded properly as a Github artifact. It appears to be the result of a minor typo in the artifact's base dir. This commit fixes the typo, and allows the artifact to be published. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Nate Rock (https://github.com/rockhowse) URL: NVIDIA/cuvs-lucene#34

Forward-merge release/26.08 into main

…vs-lucene#186) Contributes to rapidsai/build-planning#305 Proposes enforcing `yamllint` checks here. My primary motivation is to catch correctness issues in `dependencies.yaml` files, like duplicate entries silently resolving to the last one or indentation mistakes leading to filters being ignored. But this also has some side benefits for consistency, which makes it a bit easier to write automation. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#186

When `main` flipped from version 26.08 to 26.10, some CI here broke because GitHub Actions configurations had hard-coded references to container images no longer built from https://github.com/rapidsai/ci-imgs. This had to be fixed like this: * NVIDIA/cuvs-lucene#182 * NVIDIA/cuvs-lucene#179 To avoid that happening again, this proposes changes similar to NVIDIA#2340 ... dynamically generating the matrix of supported versions from the shared `rapidsai/shared-workflows` repo. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Igor Motov (https://github.com/imotov) - Bradley Dice (https://github.com/bdice) URL: NVIDIA/cuvs-lucene#189

Companion PR to [cuvs!2035](NVIDIA#2035), addresses NVIDIA/cuvs-lucene#124. Existing CAGRA search code path for each search query: - Call CAGRA search API on one index segment - Copy results back to host - Add results into host-side global top-k priority queue - Repeat for all index segments Proposed change: - Leverage new multi-segment CAGRA search API to launch all per-segment searches in one API call - Leave results on device and run GPU-accelerated select-k API to compute global top-k - Copy final top-k results to host Authors: - James Xia (https://github.com/jamxia155) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Corey J. Nolet (https://github.com/cjnolet) - Bradley Dice (https://github.com/bdice) URL: NVIDIA/cuvs-lucene#133

Forward-merge release/26.08 into main

…-to-work-with-dataset-changes Serialize CAGRA index with device-padded dataset layout

Forward-merge release/26.08 into main

imotov · 2026-08-04T22:55:59Z

/ok to test b57e9c9

imotov · 2026-08-05T01:46:04Z

/ok to test f251a6a

jameslamb

I reviewed from the narrow perspective of build/packaging (ignored any of the code in java/ that looks like it's just directly moving from the cuvs-lucene repo, like the PR description mentions).

Left a few small suggestions, but overall this looks great from that perspective.

I put a DO NOT MERGE label on this to avoid anyone accidentally squash merging this, since you mentioned you want to no-squash merge it to preserve the commit history (contribution stats, git blame, etc.). When it's ready to merge and passing CI, anyone with write access can do that by removing that label then commenting /merge nosquash.

re: your questions about codeowners... that could probably be deferred to a separate PR. At least as of this, all the changes would already be covered by these rules:

cuvs/.github/CODEOWNERS

Lines 10 to 12 in cd4bbab

    
           #java code owners 
        
           java/              @NVIDIA/cuvs-java-codeowners 
        
           examples/java/     @NVIDIA/cuvs-java-codeowners

jameslamb · 2026-08-07T15:51:41Z

          - '!ci/release/update-version.sh'
          - '!ci/run_cuvs_pytests.sh'
          - '!ci/test_java.sh'
+          - '!ci/test_lucene.sh'


Can you do another pass over these lists please?

Most of them should include both ci/build_lucene.sh and ci/test_lucene.sh, I think.

For example, this line I'm commenting on is in a group called test_cpp which controls whether cuVS's C++ tests need to run in CI. A PR that only changes ci/build_lucene.sh shouldn't at all affect the output of those tests.

I know these lists can be a little tedious to maintain, but it's really worth it... cuVS's C++ test jobs can take 3+ hours to run 😫

That makes sense, but if I understand the logic correctly test_cpp should also include exclusions for ci/build_go.sh, ci/build_java.sh, !ci/build_python.sh, !ci/build_rust.sh, shouldn't it?

…VIDIA/cuvs-lucene#194) Fixes some references in `changed-files` list in `pr.yaml`.

…VIDIA/cuvs-lucene#177) Replace the hand-rolled IVF-PQ parameter derivation and the 5M-vector algorithm switch in `CagraIndexParamsFactory` with cuVS's own heuristics: the GPU-native path now uses `AUTO_SELECT`, and the accelerated-HNSW path uses `CagraIndexParams.fromHnswParams()`, derived from `maxConn/beamWidth`. Algorithm selection now differs per path: accelerated-HNSW keeps a dataset-size crossover but at cuVS's 1M vectors rather than our 5M, while the GPU-native path's AUTO_SELECT chooses on available device memory instead of size, using each algorithm's default parameters. Expose the cuVS HNSW heuristic type on AcceleratedHNSWParams, defaulting to `SAME_GRAPH_FOOTPRINT`. Fixes NVIDIA/cuvs-lucene#149 Authors: - Igor Motov (https://github.com/imotov) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Artem M. Chirkin (https://github.com/achirkin) URL: NVIDIA/cuvs-lucene#177

cjnolet · 2026-08-08T00:28:59Z

@imotov are we planning to bring over the original history from cuVS-lucene?

Trying to understand why we are asking for no squash merge.

imotov · 2026-08-08T00:47:15Z

Yes, my preference would be to squash all of my commits related to getting this in, while leaving all historical commits that I pulled from cuvs-lucene repo untouched. This will not add too much noise from me iterating here, but will preserve all history of cuvs-lucene project. Preserving history is definitely possible, and if you check out this PR's branch and run git blame on any cuvs-lucene file, you'll see the correct commits and authors. So, as long as we don't squash this PR the history will be preserved.

The script at the bottom of the PR description is what makes this possible. It rewrites all commits in the cuvs-lucene repo as if they had originally been applied under the java/lucene directory instead of the repository root, and it adjusts all unqualified issue and PR references in commit comments to point to the cuvs-lucene repository. This ensures that files with the same name (like README.md) in both repositories retain distinct histories, and prevents clashes between issues and PRs with the same numbers in the cuvs and cuvs-lucene repositories.

imotov · 2026-08-08T00:50:36Z

/ok to test d3d8475

imotov · 2026-08-12T21:23:06Z

/ok to test 29c8862

cjnolet · 2026-08-13T11:00:31Z

+3. [Getting Started](#getting-started)
+4. [Contributing](#contributing)
+5. [References](#references)
+


Once we get cuVS-lucene settled into cuVS, we should strive to remove this readme in favor of having this content in the docs. These readmes get challenging to maintain when they are spread across various directories in the codebase.

cjnolet · 2026-08-13T11:00:56Z

@@ -0,0 +1,9 @@
+# Microbenchmarks


Same as above

cjnolet · 2026-08-13T11:02:30Z

@@ -0,0 +1,86 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.


Follow-up- this file tree / build tree is a bit awkward. Can't we just build the microbenchmarks classes into the main jar instead o having a completely separate (nested) tree for it?

cjnolet · 2026-08-13T11:03:21Z

+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0


Todo- consolidate with the top-level build.sh as a follow up.

cjnolet · 2026-08-13T11:04:14Z

@@ -0,0 +1,34 @@
+# Examples
+
+This maven project contains basic examples that showcase how `cuvs-lucene` can be used.


TODO: move to top level "examples/" (repo_root/examples/java) as a follow up

cjnolet and others added 30 commits June 5, 2025 10:12

Create LICENSE

8e3c3f0

Initial commit

8ec0201

Create .pre-commit-config.yaml

39de07f

Updates

6020469

update copyright

98d5f03

standardize license text

42e5d02

Create CONTRIBUTING.md

2b2ae08

Update README.md

cdfeca0

Adding codeowners

c5cf67b

feat: add ops-bot config

d3ff041

Update package names (NVIDIA/cuvs-lucene#9)

9ed4a68

Fixes NVIDIA/cuvs-lucene#8 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#9

Merge pull request NVIDIA/cuvs-lucene#12 from dantegd/fea-add-ci

2bbd0e3

Add CI files and workflows

Remove hnswlib index (NVIDIA/cuvs-lucene#24)

1bed6b2

Fixes NVIDIA/cuvs-lucene#7 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#24

Use main branch (NVIDIA/cuvs-lucene#32)

e3c53e3

## Description Supports rollout of new branching strategy. https://docs.rapids.ai/notices/rsn0047/ xref: rapidsai/build-planning#224

Use main shared-workflows branch (NVIDIA/cuvs-lucene#33)

d0649a7

Contributes to rapidsai/build-planning#224 ## Notes for Reviewers This is safe to admin-merge because the change is a no-op... configs on those 2 branches are identical.

GPUtester and others added 9 commits July 29, 2026 12:58

Merge pull request NVIDIA/cuvs-lucene#188 from NVIDIA/release/26.08

e5120ae

Forward-merge release/26.08 into main

Merge pull request NVIDIA/cuvs-lucene#190 from NVIDIA/release/26.08

2f2e29e

Forward-merge release/26.08 into main

Fix cuvs-lucene to work with NVIDIA#1846

3751a1b

Switch RAPIDS_BRANCH to release/26.08

153137d

Merge pull request NVIDIA/cuvs-lucene#192 from imotov/fix-cuvs-lucene…

6a17e86

…-to-work-with-dataset-changes Serialize CAGRA index with device-padded dataset layout

Merge pull request NVIDIA/cuvs-lucene#193 from NVIDIA/release/26.08

be6b4f3

Forward-merge release/26.08 into main

imotov force-pushed the cuvs-lucene-move branch 2 times, most recently from abc9903 to b57e9c9 Compare August 4, 2026 22:55

imotov marked this pull request as ready for review August 5, 2026 20:33

imotov requested review from a team as code owners August 5, 2026 20:33

imotov requested a review from jameslamb August 5, 2026 20:33

jameslamb added the DO NOT MERGE label Aug 7, 2026

jameslamb approved these changes Aug 7, 2026

View reviewed changes

jameslamb and others added 2 commits August 7, 2026 11:56

fix workflow references in changed-files list (for limiting PR CI) (N…

c21a631

…VIDIA/cuvs-lucene#194) Fixes some references in `changed-files` list in `pr.yaml`.

Igor Motov added 2 commits August 12, 2026 10:40

Merge main branch of 'cuvs-lucene' into cuvs

3d5cc04

Update cuvs-lucene build scripts

29c8862

imotov force-pushed the cuvs-lucene-move branch from d3d8475 to 29c8862 Compare August 12, 2026 21:22

cjnolet reviewed Aug 13, 2026

View reviewed changes

	#java code owners
	java/ @NVIDIA/cuvs-java-codeowners
	examples/java/ @NVIDIA/cuvs-java-codeowners

		@@ -0,0 +1,86 @@
		/*
		* SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

		@@ -0,0 +1,34 @@
		# Examples

		This maven project contains basic examples that showcase how `cuvs-lucene` can be used.

Uh oh!

Conversation

imotov commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Open questions:

Timeline for the move:

Uh oh!

imotov commented Aug 4, 2026

Uh oh!

imotov commented Aug 5, 2026

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jameslamb Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

imotov Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imotov commented Aug 8, 2026

Uh oh!

imotov commented Aug 8, 2026

Uh oh!

imotov commented Aug 12, 2026

Uh oh!

cjnolet Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

imotov commented Jul 22, 2026 •

edited

Loading

cjnolet commented Aug 8, 2026 •

edited

Loading

cjnolet Aug 13, 2026 •

edited

Loading