Repository navigation
refactor(alloc): Track objects in small extent - #9327
Open
TreeHunter9 wants to merge 2 commits into
Open
TreeHunter9 wants to merge 2 commits into
TreeHunter9 wants to merge 2 commits into
Conversation
added 2 commits
November 20, 2026 16:50
- Implement small extents cache; - Add no-wait behavior on parent allocation, so we don't have to wait parent and do allocation on its own;
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main problem
Some customers reported enormous memory usage, ~120GiB was mapped by single transaction, even though it was currently using only ~2GiB. The attachment that holds this transaction at the moment was doing basically nothing, but still, it holds 120GiB of memory.
I investigated this anomaly and discovered that all 120GiB had been taken from the parent pool (
tra_pool->att_pool). I have found that this amount of memory can be redirected only via small allocations, to be specific, when small extent need to be allocated, it can be redirected to parent pool without any limits. Looking at our allocator closely, I have found that small objects are not freed upon deallocation, these objects are only removed when the pool is destroyed. It means, until these objects is alive, we cannot give back the extents that we borrow from the parent pool.As I understand, it was done intentionally, and I see why we might want to preserve these small objects, but still, I don't think this tradeoff works in our favor. If
tra_pool->att_poolmemory usage can be fixed by commiting transaction, inatt_pool->dbb_poolchain memory can be freed only when attachment is dead, so in this non-reclamation of memory can make things even worse.Reproducible example
Solution
Before I begin my explanation, I'd like to clarify some terminology:
small extent- 4-7 KiB extent,hunkorbig extent- 64KiB extent that is allocated directly by OS. Hunk is cut into small extents, as well as extent is cut into slots. The hunk can be freed when its extents are freed, and extent can be freed when its slots are freed.I propose track when a small extents become empty, like it's done for medium objects. This will allow us to detect when a small extent contains no active objects, so we can safely deallocate it. The baseline patch is straight forward,
DoubleLinkedListwill provide policy for this logic, but overall I propose other changes to improve performance of allocator:Side note on local cache: it can lead to memory fragmentation. By memory fragmentation I mean if pool is going allocate only 8 bytes - it will hold whole 64KiB hunk. It also means if local cache will hold all 24 small extents, it is possible that each of the small extent is came from different hunk, so by holding these 24 small extents, we will hold 1.5MiB. This is still better then holding every small object forever, but this is a tradeoff between memory footprint and performance. I have introduce FIFO eviction algorithm to minimize memory footprint and limit cache size to 24 extents (I think it's a sweet spot).
Benchmarks
I'd also like to propose addition of small benchmark to Firebird project. I've read about this practice in other large projects and I think it will be a good addition for us as well. If someone also would introduce changes to allocator module, they will be able to use this bench to test their patch. For my patch, it was very helpful to see how changes affects performance in various cases.
For example, my first implementation of local cache was targeting hunk cache. Right now we have only global hunk cache, and I thought that introduction of local hunks cache would increase performance, but it didn't, so I came up with current implementation of local cache.
Benchmark can be build via
make alloc_bench.To run predefined scenarios I wrote a python script, it can be run like this:
My predefined scenarios are grouped by sequence (a lot of allocations, then all deallocation) and mixed (allocations/deallocation mixed):
Unpatched allocator → full patch
Operation latencies here cover iteration 1. Later iterations give the unpatched pool a previously allocated working set to reuse. Whole-case time includes that reuse. In eight-thread
sequence-long, mapping before the first cleanup was close (2,181 → 2,158 MiB). The deallocation time is increased because of cleanup overhead, but this overhead was basically moved from pool deletion, so caseallocation -> deallocation -> pool deletionin terms of time is the same.The large difference appeared after cleanup, this is the real problem this patch should fix. The biggest change can be seen in
sequence-longcase, where memory after cleanup reduced from 2GiB to 0.44MiB.Table
Single candidate (old behavior - 1 cache slot) → 24-slot FIFO, with parent try-lock enabled in both builds
The mixed scenario is getting improvements, as expected.
Table
Blocking parent allocation → parent try-lock, with the 24-slot FIFO in both builds
The sequence allocations are getting huge performance increase in multithreaded scenario, but memory footprint is slightly increase.
I think this is a good tradeoff.
Table
P.S.: If someone want to run these bench on it's own pc, use this suggestions: