Skip to content

Add support for Lucene 10.2–10.5 - #2702

Closed
imotov wants to merge 20 commits into
NVIDIA:mainfrom
imotov:issue-2410-support-multiple-lucene-versions
Closed

imotov wants to merge 20 commits into
NVIDIA:mainfrom
imotov:issue-2410-support-multiple-lucene-versions

Conversation

@imotov

@imotov imotov commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Build a separate jar for each Lucene minor release from 10.2 to 10.5, since Lucene changes its codec APIs between minor releases. All modules compile the same shared sources. Codecs built on an older Lucene release's default codec stay registered so existing indexes can still be read, but they can no longer write.

Fixes #2410

@copy-pr-bot

copy-pr-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@imotov imotov added improvement Improves an existing functionality Lucene labels Sep 30, 2026
Build a separate jar for each Lucene minor release from 10.2 to 10.5, since Lucene changes its codec
APIs between minor releases. All modules compile the same shared sources. Codecs built on an older Lucene release's default codec stay registered so existing indexes can still be read, but they can no longer write.

Fixes NVIDIA#2410
@imotov
imotov force-pushed the issue-2410-support-multiple-lucene-versions branch from 75b3caf to ee335be Compare September 30, 2026 02:46
@imotov

imotov commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ee335be

@imotov

imotov commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b10ab96

@imotov

imotov commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 71b6640

imotov added 11 commits October 2, 2026 06:16
Lucene 10.4+ sizes the merged graph from the M recorded in a source segment's metadata. If that M falls below maxConn, which happens for small segments where cuVS truncates the graph degree, CPU merges of such segments fail. Always record an M of at least maxConn, as Lucene's own HNSW writer does.
It will only make the same query to run twice.
@imotov

imotov commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f2db35d

@imotov

imotov commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 23004d0

@aamijar aamijar moved this to In Progress in Unstructured Data Processing Oct 22, 2026
rapids-bot Bot pushed a commit that referenced this pull request Oct 29, 2026
This PR lays the groundwork for supporting multiple Lucene versions in the same release. It doesn't add any new Lucene versions and still releases only the Lucene 10.2 jar, but it creates the infrastructure for adding new versions in the future, adds backward compatibility tests, and makes sure the jar can still be loaded on JDK 21, where it builds indexes on the CPU only.

The main code change is the replacement of the reflection-based `LuceneProvider` with the new `CuVSCodecs` class as the main entry point into the cuVS-Lucene ecosystem. `LuceneProvider` is now deprecated but will still be supported with Lucene 10.2.

The biggest structural change is that each supported Lucene version will get a separate Maven module with its version-specific pieces. This PR splits `java/cuvs-lucene/src` into `java/cuvs-lucene/src` with the shared code and `java/cuvs-lucene/lucene-10.2/src` with the Lucene 10.2-specific pieces. Every version-specific module will have its own `LuceneCompat` class, which isolates the calls whose signature or location differs between Lucene releases, so that the shared code compiles against every supported version. The artifact is now published as `cuvs-lucene-10.2`, and the old `cuvs-lucene` coordinates are relocated to it.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - https://github.com/paul-aiyedun
  - Corey J. Nolet (https://github.com/cjnolet)
  - James Lamb (https://github.com/jameslamb)

URL: #2867
rapids-bot Bot pushed a commit that referenced this pull request Nov 3, 2026
This PR adds support for Lucene 10.3.

Notes for the reviewers:
- The fern/pages/lucene_api directory is generated from sources and can be completely ignored.
- The majority of the actually interesting changes are in the java/cuvs-lucene/compat directory, which adds version-specific implementations. The way it works: each incompatible change is captured in an abstract base class that is then inherited from in the shared code. For example, KnnFloatVectorQuery introduced incompatible changes in 10.3, so there are now two abstract CompatKnnFloatVectorQuery classes, for 10.2 and 10.3, that take care of the differences, and GPUKnnFloatVectorQuery in the shared code adds all the version-independent logic. This approach is repeated for every incompatible change.
- java/cuvs-lucene/compat/README.md explains the details.
- Most of the code changes are mechanical, with a one exceptions: in 10.3, Lucene made KnnFloatVectorQuery collection "optimistic" (apache/lucene#14226), which depending on segment sizes, now breaks GPU search with k close to 1024. So all changes around MAX_CAGRA_K are a temporary workaround for that issue to make tests pass.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - James Lamb (https://github.com/jameslamb)
  - https://github.com/paul-aiyedun
  - Divye Gala (https://github.com/divyegala)

URL: #2970
rapids-bot Bot pushed a commit that referenced this pull request Nov 5, 2026
…r fallback (#3002)

Follow-up to #2970. Since Lucene 10.3, Lucene searches a filtered segment exactly when it has at most perLeafTopK candidates, a number that can be larger than k. GPUKnnFloatVectorQuery now falls back to Lucene's per-segment search when a segment has at most max(k, perLeafTopK) candidates. Since Lucene 10.2 doesn't use perLeafTopK, this doesn't change Lucene 10.2 behavior.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - https://github.com/paul-aiyedun

URL: #3002
rapids-bot Bot pushed a commit that referenced this pull request Nov 5, 2026
Follow-up to #2970. A CAGRA-only segment returns at most 1024 hits, or 512 with SINGLE_CTA.

- GPUKnnFloatVectorQuery no longer runs a filtered search whose hits Lucene would throw away because there are too few of them. If the segment has a brute-force index, the search now runs on the GPU instead of exactly on the CPU.
- The reader's new maxHits is the single place that sets the limit, and the reader now also caps SINGLE_CTA's itopk_size.
- Fixed outdated comments about k > 1024, the user guide's sentence about the limit, and MAINTAINING.md.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - https://github.com/paul-aiyedun
  - Divye Gala (https://github.com/divyegala)

URL: #3001
rapids-bot Bot pushed a commit that referenced this pull request Nov 6, 2026
Follow-up to #2970. A CAGRA-only segment returns at most 1024 hits, or 512 with SINGLE_CTA.

- GPUKnnFloatVectorQuery no longer runs a filtered search whose hits Lucene would throw away because there are too few of them. If the segment has a brute-force index, the search now runs on the GPU instead of exactly on the CPU.
- The reader's new maxHits is the single place that sets the limit, and the reader now also caps SINGLE_CTA's itopk_size.
- Fixed outdated comments about k > 1024, the user guide's sentence about the limit, and MAINTAINING.md.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - MithunR (https://github.com/mythrocks)
  - Divye Gala (https://github.com/divyegala)

URL: #3013
rapids-bot Bot pushed a commit that referenced this pull request Nov 9, 2026
Add the cuvs-lucene-10.5 module. Lucene 10.5 keeps Lucene104Codec, so it reuses the 10.4 codecs. Adapt to its API changes: mergeOneField now returns IORunnable (CompatKnnVectorsWriter variants), and the Lucene99HnswVectorsWriter constructor takes the flat format (LuceneCompat.hnswVectorsWriter).

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - Divye Gala (https://github.com/divyegala)
  - MithunR (https://github.com/mythrocks)

URL: #3021
imotov added a commit to imotov/cuvs that referenced this pull request Nov 10, 2026
The cuvs-lucene test jobs run tests from target/ directories restored from the build job. If a module's restored directory has no test classes, surefire reports it as passed without running anything. Pass -DfailIfNoTests=true so such a module fails the job instead.

Relates to NVIDIA#2702
rapids-bot Bot pushed a commit that referenced this pull request Nov 10, 2026
Adds the `cuvs-lucene-10.6` module. Lucene 10.6 keeps `Lucene104Codec` as its default codec, so the 10.4 codecs keep writing and there are no new codecs or formats. The important changes in this PR are:

- Raw vectors read hint: Lucene 10.6's flat vectors reader no longer marks its data as randomly accessed and leaves that to the format wrapping it. The cuvs-lucene formats now pass the hint through `LuceneCompat.flatReadState`, which does nothing on 10.2–10.5. Without it, searches on 10.6 would read `.vec` with normal readahead where an application configures read advice (e.g. `MMapDirectory.ADVISE_BY_CONTEXT`).
- GPU search reader merges: in 10.6 a flat reader's merge instance is a separate reader over its own sequential input. A new `knn-vectors-reader-106-merge-instance` variant has `CuVS2510GPUVectorsReader.getMergeInstance()` return a copy that reads the raw vectors through it and shares everything else with the segment's reader. It also forwards the new `getVectorCount` and `getFloatVectorValues(field, DataAccessHint)` to the flat reader.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - MithunR (https://github.com/mythrocks)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #3032
rapids-bot Bot pushed a commit that referenced this pull request Nov 10, 2026
The cuvs-lucene test jobs run tests from target/ directories restored from the build job. If a module's restored directory has no test classes, surefire reports it as passed without running anything. Pass `-DfailIfNoTests=true` so such a module fails the job instead.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #3038
imotov added a commit to imotov/cuvs that referenced this pull request Nov 10, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat).
- Class members are listed only when they are public or protected; interface members and enum constants as before.
- Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module.
- The writer compat variants document mergeOneField.

Relates to NVIDIA#2702
imotov added a commit to imotov/cuvs that referenced this pull request Nov 13, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat).
- Class members are listed only when they are public or protected; interface members and enum constants as before.
- Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module.
- The writer compat variants document mergeOneField.

Relates to NVIDIA#2702
rapids-bot Bot pushed a commit that referenced this pull request Nov 14, 2026
The Java API reference listed every member that wasn't explicitly private under "Public Members", including package-private helpers such as doMergeOneField. It also left out the members classes inherit from the hidden package-private bases (CompatKnnVectors*, BaseAcceleratedHNSWScalarQuantizedVectorsFormat).
- Class members are listed only when they are public or protected; interface members and enum constants as before.
- Classes list the documented members they inherit from a hidden base, taken from the compat variants of the newest Lucene module.
- The writer compat variants document mergeOneField.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - https://github.com/paul-aiyedun
  - Divye Gala (https://github.com/divyegala)

URL: #3042
rapids-bot Bot pushed a commit that referenced this pull request Nov 20, 2026
…earlier. (#3078)

This is a follow up for the fix in #2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU.

Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with `No growth is allowed`, or succeeds but writes a segment whose searches fail with `ArrayIndexOutOfBoundsException`. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with another `maxConn` or by Lucene's own format with another M.

- `FallbackHnswVectorsWriter` wraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs.
- New `CompatFilterKnnVectorsReader`/`Writer` compat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge.
- Tests merge segments recording a smaller and a larger M, with and without a per-field format.
- User guide troubleshooting entry, MAINTAINING.md and the compat README updated.

This works around a Lucene bug fixed in Lucene 11 so this  workaround should be removed when we add support for Lucene 11.

Relates to #2702

Authors:
  - Igor Motov (https://github.com/imotov)

Approvers:
  - https://github.com/paul-aiyedun
  - Divye Gala (https://github.com/divyegala)

URL: #3078
@imotov imotov closed this Nov 21, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Unstructured Data Processing Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improves an existing functionality Lucene non-breaking Introduces a non-breaking change

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[FEA] Support Lucene 10.2, 10.3, 10.4, 10.5 simultaneously

1 participant