Skip to content

REQ: keep the sketch valid when an allocation or an item copy throws - #587

Open
leerho wants to merge 2 commits into
masterfrom
req-update-safety
Open

leerho wants to merge 2 commits into
masterfrom
req-update-safety

Conversation

@leerho

@leerho leerho commented Nov 19, 2026

Copy link
Copy Markdown
Member

This fixes the REQ gaps found while reviewing #582, and the C++ side of the gaps that apache/datasketches-java#808 addresses in Java.

Compactor:

  • The copy constructor and the converting constructor now release their storage and the copies already made when a copy throws. The converting constructor does the same when it rejects the order of the items. Before, all of that leaked.
  • merge() first works out the sections and capacity it will need, then allocates once. Next it copies the other compactor's items, destroying the copies if one throws. Only after that does it change the state and the sections. Before, it changed the state and sections first, and it leaked the copies made before a copy that threw. It also now compares with the sketch's comparator instead of C().
  • compact() now allocates the space in the next level, and any growth that more sections need, before it flips or draws the coin or changes anything. Before, a failed compaction left the coin flipped or a random bit drawn, and it could change the sections and then fail to grow. A failed compaction now leaves both compactors as they were and draws no random bit.

Sketch:

  • update() compares the item with min and max and copies it first. It sets min and max only after the item is in level zero, so a failed append no longer leaves a min or max for an item that isn't retained.
  • merge() copies the new min and max before it sets either.

The docs of update() and merge() now say what a failure leaves behind. A comparator that throws while items are sorted or merged leaves the sketch undefined, as decided for Java in apache/datasketches-java#805.

One difference from Java: C++ never had the window in which #808's compress() could lose promoted items, because compact() already reserved space in the next level before moving any item.

Tests:

  • A compactor-level test fails each allocation of a compaction in turn. Both compactors must stay byte-identical and no random bit may be drawn. Repeating the compaction must then give exactly what it would have given without the failure.
  • Throwing item copies, made to throw at each copy in turn:
    • in updates: the sketch must be unchanged;
    • in const& merges: the target must stay consistent and usable;
    • in the copy constructor: nothing may leak.
  • A converting constructor whose comparator rejects the order must leak nothing.
  • A comparator that can't be default-constructed.

The five allocation tests fail on master and pass here. The comparator test needs this change to compile. Serialized results are byte-identical to master; I checked 600 seeded combinations of HRA and LRA, k = 4, 12 and 50, const& and rvalue merges, and double and string sketches. All tests pass under ASan and UBSan.

🤖 Generated with Claude Code

Compactor:
- The copy and converting constructors release their storage and the
  copies made when a copy throws, or when the converting constructor
  rejects the order of the items.
- merge() works out the sections and the capacity first and allocates
  once, then copies the other compactor's items and destroys the copies
  if one throws. Only then does it change the state and the sections.
  It used to change them first, and leaked the copies made before a
  copy that threw. It compares with the sketch's comparator, not C().
- compact() allocates the space in the next level, and the growth that
  more sections need, before it flips or draws the coin and changes
  anything. A failed compaction used to leave the coin flipped (or a
  random bit drawn), and could change the sections and then fail to grow.

Sketch:
- update() compares the item with min and max and copies it first, and
  sets min and max only after the item is in level zero.
- merge() copies the new min and max before it sets either.

The same gaps are addressed in datasketches-java by
apache/datasketches-java#808, except the item-loss window in compress(),
which C++ does not have: compact() reserved the next level's space before
moving any item.

Results are byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun November 19, 2026 19:02
@coveralls

coveralls commented Nov 19, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 44649722539

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.04%) to 96.924%

Details

  • Coverage increased (+0.04%) from the base build.
  • Patch coverage: 86 of 86 lines across 3 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 17460
Covered Lines: 16923
Line Coverage: 96.92%
Coverage Strength: 1886772.2 hits per line

💛 - Coveralls

A compactor that has never held an item has no storage, so serialize()
passed a null pointer with a count of 0, and the default serde passed it
on to memcpy(). UBSan on Linux reports that as undefined behavior.
Serialize the items only when there are some; the bytes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +179 to 182
if (sorted_ && !std::is_sorted(begin(), end(), comparator_)) {
destroy_items(); // the destructor doesn't run if a constructor throws
throw std::logic_error("items must be sorted");
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If comparator_ itself throws inside std::is_sorted, destroy_items() is not reached and the allocated buffer and copied items leak. Wrapping the post-copy_items check in try { ... } catch (...) { destroy_items(); throw; } avoids that.

Comment on lines 306 to +307
sort();
auto offset = hra_ ? capacity_ - num_items_ : num_items_;
auto from = hra_ ? begin() - other.get_num_items() : end();
auto to = from + other.get_num_items();
auto other_it = other.begin();
for (auto it = from; it != to; ++it, ++other_it) new (it) T(conditional_forward<FwdC>(*other_it));
if (!other.sorted_) std::sort(from, to, comparator_);
if (num_items_ > 0) std::inplace_merge(hra_ ? from : begin(), items_ + offset, hra_ ? end() : to, C());
num_items_ += other.get_num_items();
if (!other.sorted_) std::sort(from, from + num, comparator_);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since other's items are now copied into [from, from + num) before sort(), self-merging (sketch.merge(sketch)) when level 0 is unsorted sets this->sorted_ = true (aliasing other.sorted_), skips std::sort(from, from + num, comparator_), and passes an unsorted range to std::inplace_merge (sketch.merge(std::move(sketch)) also moves out of *this before merging):

req_sketch<int> s(12, true);
s.update(1);
s.update(5);
s.merge(s); // leaves [1, 5, 1, 5] with sorted_ == true

const uint32_t ne = nearest_even(ssr);
if (state_ >= static_cast<uint64_t>(1ULL << (num_sections_ - 1))
&& section_size_ > req_constants::MIN_K
if (state >= static_cast<uint64_t>(1ULL << (s.num - 1))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For k = 1024, 4 doublings reach s.num = 48 (s.size = 256), which passes check_header (num_sections <= 64). If state >= (1ULL << 47) (for example on a deserialized sketch), next_sections doubles s.num to 96 (ne = 182 >= MIN_K), and the next iteration of while (next_sections(...)) in merge() evaluates 1ULL << 95 (UB, and serialize() writes num_sections = 96 which check_header rejects). Stopping once s.num > 32 keeps s.num <= 48 consistent with check_header.

write(os, padding);
write(os, num_items_);
serde.serialize(os, begin(), num_items_);
if (num_items_ > 0) serde.serialize(os, begin(), num_items_); // an empty compactor may have no storage

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Symmetric case on the read side: deserialize_items (L518, L575) still calls alloc.allocate(0) and serde.deserialize(..., 0) when num == 0, leaving items_ != nullptr while capacity_ == 0 (and passing (nullptr, 0) to memcpy if an allocator returns nullptr for zero-sized allocations).

if (is_new_min) new_min = optional<T>(static_cast<const T&>(item));
if (is_new_max) new_max = optional<T>(static_cast<const T&>(item));
reset_sorted_view();
compactors_[0].append(std::forward<FwdT>(item));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If item aliases an element in this and compactors_[0] is at num_items_ == capacity_ (such as right after deserialize(), or after a failed compression filled the buffer), append() calls grow() and frees the backing array before constructing from item (heap-use-after-free under ASan, same pattern as #573):

req_sketch<std::string> s0(12);
s0.update(std::string(64, 'a'));
auto b = s0.serialize();
auto s = req_sketch<std::string>::deserialize(b.data(), b.size());
s.update(s.get_quantile(0.5)); // heap-use-after-free in append()

Materializing T before grow() also avoids reallocating compactors_[0] when T's copy constructor throws.

if (is_new_min) min_item_ = std::move(new_min);
if (is_new_max) max_item_ = std::move(new_max);
++num_retained_;
++n_;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should update() and merge() also cap n_ at 2^63 - 1 as in #585? Without an overflow check, ~60 repeated merges wrap n_ to 0 (is_empty() becomes true while items are retained) and reach 1ULL << 64 UB at level 64.

Comment on lines +211 to +212
if (is_new_min) min_item_ = std::move(new_min);
if (is_new_max) max_item_ = std::move(new_max);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Committing min_item_ and max_item_ before merging any compactor leaves them modified even when compactors_[0].merge() throws on allocation or on the first item copy: an empty target stays is_empty() == true while holding live min_item_ / max_item_, and a 1-item target {10} failing to merge {1, 999} ends up with n == 1, sole item 10, min == 1, max == 999. In addition, if num_levels == 1 after remove_empty_levels_above(num_levels) in catch, exact-mode serialize() omits min_item_ and max_item_, so get_min_item() / get_max_item() disagree with a serialization round-trip.

// the converting constructor checks that the sorted levels are sorted by the comparator
const req_sketch<int, ordered_by> copy(sketch, ordered_by(true));
REQUIRE(copy.get_n() == 2000);
REQUIRE(sketch.get_rank(1000) == Approx(0.5).margin(0.05));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sketch.get_CDF(...) / get_PMF(...) still fail to compile with ordered_by (and use the wrong order for stateful default-constructible comparators) because quantiles_sorted_view::check_split_points (common/include/quantiles_sorted_view.hpp:204, 213) is static and constructs Comparator() instead of using comparator_.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants