Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Build a separate jar for each Lucene minor release from 10.2 to 10.5, since Lucene changes its codec APIs between minor releases. All modules compile the same shared sources. Codecs built on an older Lucene release's default codec stay registered so existing indexes can still be read, but they can no longer write. Fixes NVIDIA#2410
imotov
force-pushed
the
issue-2410-support-multiple-lucene-versions
branch
from
September 30, 2026 02:46
75b3caf to
ee335be
Compare
Contributor
Author
|
/ok to test ee335be |
… improve robustness of builds.
…multiple-lucene-versions
Contributor
Author
|
/ok to test b10ab96 |
Contributor
Author
|
/ok to test 71b6640 |
Lucene 10.4+ sizes the merged graph from the M recorded in a source segment's metadata. If that M falls below maxConn, which happens for small segments where cuVS truncates the graph degree, CPU merges of such segments fail. Always record an M of at least maxConn, as Lucene's own HNSW writer does.
It will only make the same query to run twice.
…multiple-lucene-versions
Contributor
Author
|
/ok to test f2db35d |
Contributor
Author
|
/ok to test 23004d0 |
Contributor
Author
rapids-bot Bot
pushed a commit
that referenced
this pull request
Oct 29, 2026
This PR lays the groundwork for supporting multiple Lucene versions in the same release. It doesn't add any new Lucene versions and still releases only the Lucene 10.2 jar, but it creates the infrastructure for adding new versions in the future, adds backward compatibility tests, and makes sure the jar can still be loaded on JDK 21, where it builds indexes on the CPU only. The main code change is the replacement of the reflection-based `LuceneProvider` with the new `CuVSCodecs` class as the main entry point into the cuVS-Lucene ecosystem. `LuceneProvider` is now deprecated but will still be supported with Lucene 10.2. The biggest structural change is that each supported Lucene version will get a separate Maven module with its version-specific pieces. This PR splits `java/cuvs-lucene/src` into `java/cuvs-lucene/src` with the shared code and `java/cuvs-lucene/lucene-10.2/src` with the Lucene 10.2-specific pieces. Every version-specific module will have its own `LuceneCompat` class, which isolates the calls whose signature or location differs between Lucene releases, so that the shared code compiles against every supported version. The artifact is now published as `cuvs-lucene-10.2`, and the old `cuvs-lucene` coordinates are relocated to it. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - https://github.com/paul-aiyedun - Corey J. Nolet (https://github.com/cjnolet) - James Lamb (https://github.com/jameslamb) URL: #2867
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 3, 2026
This PR adds support for Lucene 10.3. Notes for the reviewers: - The fern/pages/lucene_api directory is generated from sources and can be completely ignored. - The majority of the actually interesting changes are in the java/cuvs-lucene/compat directory, which adds version-specific implementations. The way it works: each incompatible change is captured in an abstract base class that is then inherited from in the shared code. For example, KnnFloatVectorQuery introduced incompatible changes in 10.3, so there are now two abstract CompatKnnFloatVectorQuery classes, for 10.2 and 10.3, that take care of the differences, and GPUKnnFloatVectorQuery in the shared code adds all the version-independent logic. This approach is repeated for every incompatible change. - java/cuvs-lucene/compat/README.md explains the details. - Most of the code changes are mechanical, with a one exceptions: in 10.3, Lucene made KnnFloatVectorQuery collection "optimistic" (apache/lucene#14226), which depending on segment sizes, now breaks GPU search with k close to 1024. So all changes around MAX_CAGRA_K are a temporary workaround for that issue to make tests pass. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - James Lamb (https://github.com/jameslamb) - https://github.com/paul-aiyedun - Divye Gala (https://github.com/divyegala) URL: #2970
This was referenced Nov 3, 2026
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 5, 2026
…r fallback (#3002) Follow-up to #2970. Since Lucene 10.3, Lucene searches a filtered segment exactly when it has at most perLeafTopK candidates, a number that can be larger than k. GPUKnnFloatVectorQuery now falls back to Lucene's per-segment search when a segment has at most max(k, perLeafTopK) candidates. Since Lucene 10.2 doesn't use perLeafTopK, this doesn't change Lucene 10.2 behavior. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - https://github.com/paul-aiyedun URL: #3002
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 5, 2026
Follow-up to #2970. A CAGRA-only segment returns at most 1024 hits, or 512 with SINGLE_CTA. - GPUKnnFloatVectorQuery no longer runs a filtered search whose hits Lucene would throw away because there are too few of them. If the segment has a brute-force index, the search now runs on the GPU instead of exactly on the CPU. - The reader's new maxHits is the single place that sets the limit, and the reader now also caps SINGLE_CTA's itopk_size. - Fixed outdated comments about k > 1024, the user guide's sentence about the limit, and MAINTAINING.md. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - https://github.com/paul-aiyedun - Divye Gala (https://github.com/divyegala) URL: #3001
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 6, 2026
Follow-up to #2970. A CAGRA-only segment returns at most 1024 hits, or 512 with SINGLE_CTA. - GPUKnnFloatVectorQuery no longer runs a filtered search whose hits Lucene would throw away because there are too few of them. If the segment has a brute-force index, the search now runs on the GPU instead of exactly on the CPU. - The reader's new maxHits is the single place that sets the limit, and the reader now also caps SINGLE_CTA's itopk_size. - Fixed outdated comments about k > 1024, the user guide's sentence about the limit, and MAINTAINING.md. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - MithunR (https://github.com/mythrocks) - Divye Gala (https://github.com/divyegala) URL: #3013
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 9, 2026
Add the cuvs-lucene-10.5 module. Lucene 10.5 keeps Lucene104Codec, so it reuses the 10.4 codecs. Adapt to its API changes: mergeOneField now returns IORunnable (CompatKnnVectorsWriter variants), and the Lucene99HnswVectorsWriter constructor takes the flat format (LuceneCompat.hnswVectorsWriter). Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - Divye Gala (https://github.com/divyegala) - MithunR (https://github.com/mythrocks) URL: #3021
imotov
added a commit
to imotov/cuvs
that referenced
this pull request
Nov 10, 2026
The cuvs-lucene test jobs run tests from target/ directories restored from the build job. If a module's restored directory has no test classes, surefire reports it as passed without running anything. Pass -DfailIfNoTests=true so such a module fails the job instead. Relates to NVIDIA#2702
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 10, 2026
Adds the `cuvs-lucene-10.6` module. Lucene 10.6 keeps `Lucene104Codec` as its default codec, so the 10.4 codecs keep writing and there are no new codecs or formats. The important changes in this PR are: - Raw vectors read hint: Lucene 10.6's flat vectors reader no longer marks its data as randomly accessed and leaves that to the format wrapping it. The cuvs-lucene formats now pass the hint through `LuceneCompat.flatReadState`, which does nothing on 10.2–10.5. Without it, searches on 10.6 would read `.vec` with normal readahead where an application configures read advice (e.g. `MMapDirectory.ADVISE_BY_CONTEXT`). - GPU search reader merges: in 10.6 a flat reader's merge instance is a separate reader over its own sequential input. A new `knn-vectors-reader-106-merge-instance` variant has `CuVS2510GPUVectorsReader.getMergeInstance()` return a copy that reads the raw vectors through it and shares everything else with the segment's reader. It also forwards the new `getVectorCount` and `getFloatVectorValues(field, DataAccessHint)` to the flat reader. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - MithunR (https://github.com/mythrocks) - Dante Gama Dessavre (https://github.com/dantegd) URL: #3032
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 10, 2026
The cuvs-lucene test jobs run tests from target/ directories restored from the build job. If a module's restored directory has no test classes, surefire reports it as passed without running anything. Pass `-DfailIfNoTests=true` so such a module fails the job instead. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - James Lamb (https://github.com/jameslamb) URL: #3038
imotov
added a commit
to imotov/cuvs
that referenced
this pull request
Nov 10, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat). - Class members are listed only when they are public or protected; interface members and enum constants as before. - Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module. - The writer compat variants document mergeOneField. Relates to NVIDIA#2702
imotov
added a commit
to imotov/cuvs
that referenced
this pull request
Nov 13, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat). - Class members are listed only when they are public or protected; interface members and enum constants as before. - Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module. - The writer compat variants document mergeOneField. Relates to NVIDIA#2702
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 14, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat). - Class members are listed only when they are public or protected; interface members and enum constants as before. - Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module. - The writer compat variants document mergeOneField. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - https://github.com/paul-aiyedun - Divye Gala (https://github.com/divyegala) URL: #3042
rapids-bot Bot
pushed a commit
that referenced
this pull request
Nov 20, 2026
…earlier. (#3078) This is a follow up for the fix in #2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU. Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with `No growth is allowed`, or succeeds but writes a segment whose searches fail with `ArrayIndexOutOfBoundsException`. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with another `maxConn` or by Lucene's own format with another M. - `FallbackHnswVectorsWriter` wraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs. - New `CompatFilterKnnVectorsReader`/`Writer` compat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge. - Tests merge segments recording a smaller and a larger M, with and without a per-field format. - User guide troubleshooting entry, MAINTAINING.md and the compat README updated. This works around a Lucene bug fixed in Lucene 11 so this workaround should be removed when we add support for Lucene 11. Relates to #2702 Authors: - Igor Motov (https://github.com/imotov) Approvers: - https://github.com/paul-aiyedun - Divye Gala (https://github.com/divyegala) URL: #3078
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build a separate jar for each Lucene minor release from 10.2 to 10.5, since Lucene changes its codec APIs between minor releases. All modules compile the same shared sources. Codecs built on an older Lucene release's default codec stay registered so existing indexes can still be read, but they can no longer write.
Fixes #2410