Skip to content

Fix CPU fallback merges of segments written by cuvs-lucene 26.10 and earlier. - #3078

Merged
rapids-bot[bot] merged 4 commits into
NVIDIA:release/26.12from
imotov:issue-2702-fix-for-indices-created-with-wrong-m
Nov 20, 2026
Merged

rapids-bot[bot] merged 4 commits into
NVIDIA:release/26.12from
imotov:issue-2702-fix-for-indices-created-with-wrong-m

Conversation

@imotov

@imotov imotov commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This is a follow up for the fix in #2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU.

Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with No growth is allowed, or succeeds but writes a segment whose searches fail with ArrayIndexOutOfBoundsException. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with another maxConn or by Lucene's own format with another M.

  • FallbackHnswVectorsWriter wraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs.
  • New CompatFilterKnnVectorsReader/Writer compat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge.
  • Tests merge segments recording a smaller and a larger M, with and without a per-field format.
  • User guide troubleshooting entry, MAINTAINING.md and the compat README updated.

This works around a Lucene bug fixed in Lucene 11 so this workaround should be removed when we add support for Lucene 11.

Relates to #2702

…earlier.

This is a follow up for the fix in NVIDIA#2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU.

Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with `No growth is allowed`, or succeeds but writes a segment whose searches fail with `ArrayIndexOutOfBoundsException`. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with another `maxConn` or by Lucene's own format with another M.

- `FallbackHnswVectorsWriter` wraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs.
- New `CompatFilterKnnVectorsReader`/`Writer` compat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge.
- Tests merge segments recording a smaller and a larger M, with and without a per-field format.
- User guide troubleshooting entry, MAINTAINING.md and the compat README updated.

This works around a Lucene bug fixed in Lucene 11 so this  workaround should be removed when we add support for Lucene 11.

Relates to NVIDIA#2702
@imotov imotov self-assigned this Nov 16, 2026
@imotov imotov added bug Something isn't working non-breaking Introduces a non-breaking change Lucene labels Nov 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@imotov

imotov commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e6bd172

@imotov
imotov marked this pull request as ready for review November 17, 2026 18:34
@imotov
imotov requested review from a team as code owners November 17, 2026 18:34
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved CPU fallback handling when merging HNSW indexes whose graphs use different connection settings, helping preserve vector access and produce compatible merged graphs.
  • Documentation
    • Added guidance on HNSW merge compatibility across Lucene releases, including steps for switching formats and handling indexes that may need rewriting or rebuilding.
    • Updated compatibility notes for supported Lucene versions and refreshed API source links. Clarified merge behavior and format-switching considerations for existing indexes.

The PR adds Lucene reader and writer compatibility wrappers and a CPU fallback writer for HNSW merges with mismatched graph M values. It adds merge tests and troubleshooting guidance, maintenance documentation, and updates source links in API documentation.

Changes: Lucene HNSW merge compatibility

Layer / File(s) Summary
Forward reader and writer operations
java/cuvs-lucene/src/main/java/com/nvidia/cuvs/lucene/FilterKnnVectorsReader.java, java/cuvs-lucene/src/main/java/com/nvidia/cuvs/lucene/FilterKnnVectorsWriter.java, java/cuvs-lucene/compat/knn-vectors-reader-*/..., java/cuvs-lucene/compat/knn-vectors-writer-*/..., java/cuvs-lucene/compat/README.md, java/cuvs-lucene/README.md, java/cuvs-lucene/MAINTAINING.md
Reader and writer wrappers forward operations to wrapped instances. Release-specific adapters forward methods with differing Lucene signatures. The documentation describes forwarded methods and Lucene defaults.
Handle mismatched graph M during CPU merges
java/cuvs-lucene/src/main/java/com/nvidia/cuvs/lucene/FallbackHnswVectorsWriter.java, java/cuvs-lucene/src/main/java/com/nvidia/cuvs/lucene/*Accelerated*VectorsFormat.java, java/cuvs-lucene/MAINTAINING.md
The fallback writer hides nonempty source graphs whose M differs from maxConn while Lucene builds the merged graph, then restores the source readers. Accelerated formats use this writer when cuVS is unsupported.
Test merges and update Lucene documentation
java/cuvs-lucene/src/test/java/com/nvidia/cuvs/lucene/TestQuantizedFallbackWriters.java, fern/pages/user_guide/lucene.md, fern/pages/lucene_api/*
Tests cover CPU fallback merges with source M values smaller or larger than the target, including mixed values. The user guide adds merge troubleshooting guidance, and API source links point to updated Java line numbers.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: paul-aiyedun

Merge Risk: Minimal · up to 8f34f

This change makes CPU fallback merges of older cuvs-lucene segments with a mismatched HNSW M insert the vectors rather than merge the old graphs, which avoids Lucene rejecting or corrupting the merge. Tests for smaller and larger source M are described, and no unresolved concerns remain. The change looks ready to merge.

Security architecture risk: Minimal · up to ba165

The inspected change addresses merge compatibility without establishing a new security exposure or weakened ownership guarantee. Reader substitutions are restored after graph building and preparation failures, while document filters and integrity checks remain delegated.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • observed — The inspected changed operation acts on source-segment readers in the caller-supplied MergeState and the destination writer constructed from SegmentWriteState. Service, tenant and deployment exposure beyond this index operation was not supplied.

Trust Boundaries and Controls

  • inferred — Source graph metadata influences whether graphs are reused or vectors are reinserted within the existing merge. The inspected adapters do not turn that choice into additional authority or remove the document-filter and integrity controls delegated to the underlying reader.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: CPU fallback merges of segments written by cuvs-lucene 26.10 and earlier. The changeset adds FallbackHnswVectorsWriter, which hides source graphs with a different M du…
Description check Passed The description states the problem (segments with a different HNSW M breaking CPU fallback merges after upgrade to Lucene 10.3+) and describes the matching changes: FallbackHnswVectorsWriter, the Comp…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: CPU fallback merges of segments written by cuvs-lucene 26.10 and earlier. The changeset adds FallbackHnswVectorsWriter, which hides source graphs with a different M during merge, plus compat reader/writer variants, tests, and docs for that behavior. The title is specific and matches the primary intent.

Full details: Description check

Explanation

The description states the problem (segments with a different HNSW M breaking CPU fallback merges after upgrade to Lucene 10.3+) and describes the matching changes: FallbackHnswVectorsWriter, the CompatFilterKnnVectorsReader/Writer variants, merge tests, and doc updates. It is directly related to the changeset.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ba165bd and 8f34f2b.

Files selected for processing: 2

  • java/cuvs-lucene/MAINTAINING.md
  • java/cuvs-lucene/README.md
Files skipped from review as they are similar to previous changes: 2

  • java/cuvs-lucene/MAINTAINING.md
  • java/cuvs-lucene/README.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@paul-aiyedun paul-aiyedun left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes overall look good to me. However, I had some questions and a couple of comments about possible code duplication.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There appears to be duplication across the CompatFilterKnnVectorsReader classes. Can we move shared methods to a common base class?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment to the above about moving shared methods to a common base class.

}

/** Returns the most neighbors that a node of {@code graph} has, on any level. */
private static int mostNeighbors(HnswGraph graph) throws IOException {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a distinction between the level 0 node count and that of higher levels?

.setCodec(factory.codecWith().apply(sourceParams))
.setMaxBufferedDocs(OTHER_M_SEGMENT_DOCS)
.setMergePolicy(NoMergePolicy.INSTANCE);
try (IndexWriter writer = new IndexWriter(dir, sourceConfig)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider having the test mix segments with correct and incorrect max conn values?

@imotov

imotov commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 1b01079 into NVIDIA:release/26.12 Nov 20, 2026
74 checks passed
@github-project-automation github-project-automation Bot moved this from Approved to Done in Unstructured Data Processing Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Lucene non-breaking Introduces a non-breaking change

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants