Repository navigation
Fix CPU fallback merges of segments written by cuvs-lucene 26.10 and earlier. - #3078
Conversation
…earlier. This is a follow up for the fix in NVIDIA#2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU. Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with `No growth is allowed`, or succeeds but writes a segment whose searches fail with `ArrayIndexOutOfBoundsException`. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with another `maxConn` or by Lucene's own format with another M. - `FallbackHnswVectorsWriter` wraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs. - New `CompatFilterKnnVectorsReader`/`Writer` compat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge. - Tests merge segments recording a smaller and a larger M, with and without a per-field format. - User guide troubleshooting entry, MAINTAINING.md and the compat README updated. This works around a Lucene bug fixed in Lucene 11 so this workaround should be removed when we add support for Lucene 11. Relates to NVIDIA#2702
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test e6bd172 |
|
No actionable comments were generated in the recent review. Summary
Merge Risk:
|
paul-aiyedun
left a comment
There was a problem hiding this comment.
Changes overall look good to me. However, I had some questions and a couple of comments about possible code duplication.
There was a problem hiding this comment.
There appears to be duplication across the CompatFilterKnnVectorsReader classes. Can we move shared methods to a common base class?
There was a problem hiding this comment.
Similar comment to the above about moving shared methods to a common base class.
| } | ||
|
|
||
| /** Returns the most neighbors that a node of {@code graph} has, on any level. */ | ||
| private static int mostNeighbors(HnswGraph graph) throws IOException { |
There was a problem hiding this comment.
Should there be a distinction between the level 0 node count and that of higher levels?
| .setCodec(factory.codecWith().apply(sourceParams)) | ||
| .setMaxBufferedDocs(OTHER_M_SEGMENT_DOCS) | ||
| .setMergePolicy(NoMergePolicy.INSTANCE); | ||
| try (IndexWriter writer = new IndexWriter(dir, sourceConfig)) { |
There was a problem hiding this comment.
Did you consider having the test mix segments with correct and incorrect max conn values?
…-fix-for-indices-created-with-wrong-m
|
/merge |
This is a follow up for the fix in #2769. While we fixed the bug back then, any segments written before this fix can potentially break index after upgrade to Lucene 10.3+, if such merge is perfromed on CPU.
Lucene's HNSW writer, which the accelerated formats use as their CPU fallback, cannot merge segments whose graph records an HNSW M other than its own. Depending on the Lucene release, the merge fails with
No growth is allowed, or succeeds but writes a segment whose searches fail withArrayIndexOutOfBoundsException. cuvs-lucene 26.10 and earlier recorded such an M, and so do segments written with anothermaxConnor by Lucene's own format with another M.FallbackHnswVectorsWriterwraps Lucene's writer. While a field's graph is merged, it hides the graphs of source segments that record another M, so Lucene inserts their vectors instead of starting from those graphs.CompatFilterKnnVectorsReader/Writercompat variants (10.2–10.6) forward to the wrapped reader or writer. On 10.5+ they wrap the deferred graph merge.This works around a Lucene bug fixed in Lucene 11 so this workaround should be removed when we add support for Lucene 11.
Relates to #2702