Skip to content

KLL: keep the sketch valid when an item copy throws; cap n at 2^63 - 1 - #585

Open
leerho wants to merge 1 commit into
masterfrom
kll-merge-copy-safety
Open

leerho wants to merge 1 commit into
masterfrom
kll-merge-copy-safety

Conversation

@leerho

@leerho leerho commented Nov 19, 2026

Copy link
Copy Markdown
Member

Follow-up to tisonkun's review of #582.

When both sketches held items at the same level, the merge copied other's items after this sketch's items had been moved out. A copy that threw left destroyed items counted, so the destructor destroyed them again. A weighted update reaches this once the sketch is in estimation mode, and merge(const&) reached it before.

Changes:

  • populate_work_arrays() copies (or moves) other's items into the work buffer first. If a copy throws, it destroys them, and this sketch is unchanged. Then this sketch's items are merged in front of them using moves only. A new helper, merge_sorted_arrays_into_gap(), replaces the version that copied from const T*.
  • update() and merge() construct an item in level zero before counting it, and reset the sorted view before any change. If a copy throws during a merge, the sketch stays valid and usable but may hold part of the other sketch. This is documented.
  • The copy constructor and the converting constructor release their items when a copy or the sort check throws.
  • kll_helper uses the sketch's comparator instead of C(), so a stateful comparator is respected, and a comparator without a default constructor works.
  • n is capped at kll_constants::MAX_N (2^63 - 1), as in datasketches-java. Updates and merges that would pass it throw std::overflow_error and leave the sketch unchanged. This also keeps a merge from reaching the level count that made level_capacity throw partway through.
  • The doc comment for update(item, weight) now describes both paths. It notes that a comparator that throws leaves the sketch undefined, the same decision as for Java classic quantiles in Quantiles items: keep the sketch consistent when an update throws datasketches-java#805.

Tests:

  • tisonkun's fixture: k=8, 9 updates, weight 18.
  • Failure at each item copy of a merge(const&) with overlapping levels.
  • Failure at each copy of the copy constructor.
  • A converting constructor whose comparator rejects the order.
  • A comparator that can't be default-constructed.
  • The n cap, for updates and merges.
  • The "weighted update: estimation mode" section is now seeded.

The four allocation tests fail on master under ASan/UBSan and pass here. Serialized merge results are byte-identical to master; I checked k = 8, 20 and 200 for float and string sketches, const& and rvalue merges, and weighted updates. All tests pass under ASan+UBSan.

🤖 Generated with Claude Code

Merging items at levels where both sketches hold items copied from other
after this sketch's items had been moved out, so a throwing copy left
destroyed items counted (a double free). A weighted update reaches this
once the sketch is in estimation mode.

- populate_work_arrays() copies (or moves) other's items first; if a copy
  throws, it destroys them and this sketch is unchanged. Then this sketch's
  items are merged in front of them with moves only.
- update() and merge() construct an item in level zero before counting it,
  and reset the sorted view before any change.
- The copy and converting constructors release their items when a copy
  or the sort check throws.
- The KLL helpers use the sketch's comparator instead of C().
- n is capped at kll_constants::MAX_N (2^63 - 1), as in datasketches-java.
  This also keeps a merge from reaching 62 levels.
- Docs: the cost of a weighted update with a small weight.
- Tests: tisonkun's fixture from #582, failure at each copy of a merge and
  of the copy constructor, a converting constructor that rejects the
  order, a comparator that can't be default-constructed, the n cap; seed
  the "weighted update: estimation mode" section.

Merge results are byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun November 19, 2026 17:49
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 44581048119

Coverage increased (+0.01%) to 96.898%

Details

  • Coverage increased (+0.01%) from the base build.
  • Patch coverage: 65 of 65 lines across 2 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 17438
Covered Lines: 16897
Line Coverage: 96.9%
Coverage Strength: 1881465.24 hits per line

💛 - Coveralls

@tisonkun tisonkun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants