Fix GPU and Java-heap OOM when building large vector indexes

[nvzm123]

Summary

This ports NVIDIA/cuvs-lucene#173 into java/cuvs-lucene following the move into the cuVS monorepo.

It addresses two memory-pressure problems when building accelerated HNSW indexes over large datasets:

• Uses host-backed cuVS matrices instead of eagerly placing the complete dataset in device memory.
• Streams merged vectors into a native host matrix and samples higher-layer vectors from that matrix, avoiding a complete List<float[]> copy on the Java heap.

The change also consolidates the HNSW graph and field-writing paths around CuVSMatrix, removes redundant size/resource parameters, and derives quantized-vector byte width from the matrix so binary and scalar quantization use the correct width.

The original work superseded NVIDIA/cuvs-lucene#141. The related increase of MAX_HNSW_LAYERS to 99 is already present on the current monorepo main and is therefore not part of this diff.

Validation status

Fresh port-only structural checks:

• cd java/cuvs-lucene && mvn -q -DskipTests compile — passed
• cd java/cuvs-lucene && mvn -q spotless:check — passed

These commands compile and check formatting; they do not constitute behavioral or GPU validation.

The source PR historically reported local passes for:

• TestMerge, including testLargeScaleMerge with -DlargeScale=true
• TestCuVSVectorsFormat
• TestLucene99AcceleratedHNSWVectorsFormat
• TestQuantizedVectorsFormats

Those results predate this port and are not validation of the current branch. The original PR's final standalone-repository CI run was not green, so the relevant Java/GPU suites must be rerun from java/cuvs-lucene. We also need combined validation with PR #2475, including its full PyLucene GPU end-to-end suite.

The source PR also recorded an intermittent, seed-dependent TestLucene99AcceleratedHNSWVectorsFormat.testCheckIntegrityReadsAllBytes failure during full-suite randomized graph testing. It passed in isolation and reportedly occurred at a similar rate on the source base branch. This historical observation has not yet been re-evaluated in the monorepo.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.