Skip to content

Classic quantiles: keep the sketch valid when an update or merge throws - #586

Open
leerho wants to merge 1 commit into
masterfrom
quantiles-update-safety
Open

leerho wants to merge 1 commit into
masterfrom
quantiles-update-safety

Conversation

@leerho

@leerho leerho commented Nov 19, 2026

Copy link
Copy Markdown
Member

update() counted the item in n before processing a full base buffer, and that processing allocates the next level. If the allocation failed, the base buffer stayed at 2k items and was never compacted again. From then on the base buffer grew without bound, and serialize() to bytes threw. This affected every item type, double included. The merges had similar gaps when an allocation failed or an item's copy threw.

This follows apache/datasketches-java#805, which fixed the same flaw in Java's QuantilesItemsSketch.

Changes to update():

  • It compares the item with min and max, copies it if it is a new min or max, and allocates anything the base buffer and levels need, all before changing the sketch.
  • It commits n, min and max only after the full base buffer has been processed.
  • If an allocation or a copy throws, the sketch is unchanged.

Changes to the merges:

  • They allocate the target's levels, base buffer and scratch buffers first, so that an allocation failure leaves the target unchanged.
  • They set min and max first, then advance n with each level they merge. A copy that throws from a const source then leaves a valid sketch that may hold part of the other one.
  • A level that was only partly copied is cleared again.
  • When the source is an rvalue, items are moved instead of copied. This includes the tails in merge_two_size_k_buffers(), which copied even before.
  • The paths that move this sketch's items into a copy of the other sketch now do all their allocations before the first move.
  • A failed update or merge removes the levels it added, since the iterator expects none above the highest level in use.
  • After levels_ grows, empty levels get back their capacity of k. With an allocator whose moves can throw, growing levels_ copies the levels, and a copy of an empty level has no capacity. Before this change, that only went unnoticed because of the order of the steps.

A comparator that throws while items are sorted or merged, or that is inconsistent, leaves the sketch undefined. That matches the decision for Java in #805, and the docs of update() and merge() now say so. Java sorts a copy of the base buffer so that a throwing comparator leaves it as it was. In C++ that would copy every item of the base buffer, so this PR doesn't.

optional::emplace() (in common) now forwards its arguments. Before, it copied when it should have moved, while declaring itself noexcept for a move. A copy that threw therefore called std::terminate, and that affected every sketch that uses it.

Tests, in a new quantiles_sketch_allocation_test.cpp:

  • Allocation failures in updates. Fail each allocation of a run of updates in turn. The sketch must then be byte-identical to one that never saw the failed item.
  • Allocation failures in merges. The same for every merge path, through both const& and rvalue merges. The target must be unchanged, and repeating the merge must give the same bytes as a merge that never failed.
  • Throwing copies. Make each item copy throw in turn during updates and const& merges. The sketch must stay consistent and usable, with no leaks.

All four fail on master and pass here. Serialized results are byte-identical to master; I checked 1,296 seeded merges with k = 4, 16 and 128, covering all merge paths, const& and rvalue merges, and double and string sketches. All tests pass under ASan and UBSan.

🤖 Generated with Claude Code

update() counted the item in n before processing a full base buffer, and
processing allocated the next level. If that allocation failed, the base
buffer stayed at 2k items and was never compacted again: memory then grew
without bound, and serialize() to bytes threw.

update() now compares and copies first, allocates everything next, and
commits n, min and max only after the full base buffer is processed, as
datasketches-java does since apache/datasketches-java#805.

Merges:
- allocate the target's levels, base buffer and scratch buffers before
  changing the target, so that an allocation failure leaves it unchanged;
- set min and max first, then advance n with each level, so that a copy
  that throws from a const source leaves a valid sketch;
- undo a partly copied level;
- move instead of copy when the source is an rvalue, including the tails
  in merge_two_size_k_buffers();
- no longer move this sketch's items into a copy of the other sketch
  before the allocations that could fail;
- remove levels that a failed update or merge added (the iterator expects
  none above the highest level in use);
- restore the capacity of empty levels after levels_ grows: with an
  allocator whose moves can throw, growing levels_ copies the levels.

A comparator that throws while items are sorted or merged leaves the
sketch undefined, as decided for Java; the docs say so.

optional::emplace() now forwards its arguments. It copied instead of
moving while declaring noexcept for a move, so a copy that threw called
std::terminate.

Results are byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun November 19, 2026 19:02
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 44594066230

Coverage increased (+0.06%) to 96.948%

Details

  • Coverage increased (+0.06%) from the base build.
  • Patch coverage: 1 uncovered change across 1 file (92 of 93 lines covered, 98.92%).
  • 5 coverage regressions across 2 files.

Uncovered Changes

File Changed Covered %
quantiles/include/quantiles_sketch_impl.hpp 92 91 98.91%
Total (2 files) 93 92 98.92%

Coverage Regressions

5 previously-covered lines in 2 files lost coverage.

File Lines Losing Coverage Coverage
quantiles/include/quantiles_sketch_impl.hpp 3 95.52%
common/include/serde.hpp 2 98.02%

Coverage Stats

Coverage Status
Relevant Lines: 17434
Covered Lines: 16902
Line Coverage: 96.95%
Coverage Strength: 1882584.17 hits per line

💛 - Coveralls

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants