Repository navigation
Conversation
Replace the mutex-guarded 64-entry SortedArray in every RelationPages with a single 1024-entry hash + CLOCK cache owned by the attachment. Each RelationPages incarnation gets a process-wide monotonic 64-bit PageSetId (regenerated on reset and reuse, preserved across retaining commits), preventing stale-key collisions when relation, tablespace or LTT IDs are reused. Cached entries are explicitly purged from the current attachment's cache on page deletion, validation failure, and page-set reset/reuse.
hvlad
left a comment
There was a problem hiding this comment.
I found no explicit problems with the code, please consider few notes that I've made.
Results looks good!
|
Is it necessary to have in the key all three |
The |
|
Why it isn't called "pageNumber" then...? |
Because |
Modern IDEs let us have |
|
Spam again, nothing new |
|
Sure, readability of code is an old thing that always was ignored by the team. |
NB: This description (AI assisted, manually validated) is intentionally left quite verbose as it documents some decisions that may be not obvious from the patch itself. My apologies for the long read.
Every
RelationPagesobject carries aSortedArrayof up to 64{sequence → physical number}entries:5.0-release kept the same map but without the lock, which is why it scales far better under concurrency. The lock is also visible on a single thread: the old map pays for a lock/unlock pair, a binary search and mark maintenance on every data page lookup, while the new cache is one lock-free hash probe.
Design
One
DataPageNumberCacheper attachment, created lazily inatt_pooland freed with it. It embeds 1024 preallocated entries (m_items), hashes into 1021 buckets (m_map) and uses clock (second-chance) replacement, so recently used entries survive while a single cold entry is replaced per miss. Cached numbers are re-validated before use, so a stale or evicted entry can only cost a pointer page lookup, never return wrong data. Given the cache is attachment-local, the engine's per-attachment serialization is sufficient.Why
RelationPageskeeps itsgetDPNumber/setDPNumberwrappersThe cache could have been called directly from
dpm.epp, but the wrappers were kept deliberately:RelationPages, obtained fromgetPages(),getBasePages()orgetTempPages(), and need its page space for the window anyway — routing through the object avoids resolving the page set or deriving its identity a second time;m_pageSetIdlives inRelationPages, sogetPageSetId()belongs next to it; the attachment only owns the storage;RelationPagesonly;constinRelation.h, so nothing is added to the hot path — they compile intoDPM_get/DPM_next.Why the cache class has its own header
Attachment.his one of the most widely included engine headers and does not includeRelation.h, whileRelation.hincludesAttachment.h. KeepingDataPageNumberCachein its own header (src/jrd/DataPageNumberCache.h), with noJrddependencies beyondfirebird.handHash.hand withPageSetIddefined as a plain value type (FB_UINT64), avoids an include cycle betweenAttachment.handRelation.h, keepsAttachment.hself-contained, and lets the unit tests exercise the cache without anAttachmentor a database.Why this key format
A key must uniquely identify a specific incarnation of a page set:
RelationPages(a tablespace move creates a new page space, GTT/LTT instances have separate page sets), andRelationPagesobjects are recycled from a free list;{relationId, pageSpaceId, instanceId}is not unique across incarnations:ALTER TABLE ... SET TABLESPACEround-trips (PRIMARY → TS → PRIMARY) reuse{relationId, DB_PAGE_SPACE, 0}on the sameRelationPagesviareset(pageSpaceId);DROP TABLEfollowed byCREATE TABLEreuses deleted relation IDs (RelationNode::generateRelIdinDdlNodes.epp:6736-6838), producing the same{relationId, DB_PAGE_SPACE, 0}in a newly allocatedRelationPages;DECLARE LOCAL TEMPORARY TABLE) have a deterministicrelationId(MIN_DECLARED_LTT_ID + tableNumber) and reuse the request'sreq_local_table_instance_idacross repeated request executions afterDeclareLocalTableNode::reset()frees and recycles the page set viareuse();att_next_ltt_idacross 1024 IDs (DdlNodes.epp:10370-10393), and tablespace IDs wrap modulo 255 (TablespaceNodes.epp:208-211).Because
PAG_release_pagesonly flips PIP bits (andCCH_clean_pageonly clears dirty flags for temp pages) without zeroing the on-pagedata_pageheader, an unallocated old data page still carriespag_type == pag_data,dpg_relation == relationIdanddpg_sequence == dpSequence, so header validation (pageOk) cannot distinguish a reused semantic tuple from the previous incarnation.Each page-set incarnation therefore gets a process-wide monotonic 64-bit ID, backed by a static atomic counter:
m_pageSetIdis assigned in theRelationPagesconstructor and when the page set is reset or recycled (updatePageSpaceIdinreset()andupdateIdentityinreuse());setInstanceId()(called byretainPages()on a retaining commit) keepsm_pageSetIdunchanged because the physical pages are preserved;reset(),reuse(),SET TABLESPACEround-trips, LTT re-executions, relation/tablespace ID reuse, or memory-pool address reuse after aRelationPagesis destroyed and reallocated;m_pageSetIdis stored asstd::atomic<PageSetId>and read withstd::memory_order_relaxedingetPageSetId(), so lock-free readers ingetDPNumber/setDPNumbercannot race or tear against a concurrentreset().Key variants considered during implementation and measured on the scan benchmark:
{RelationPages*, generation}(prototype){relationId, pageSpaceId, instanceId, sequence}{relationId, tagged id, sequence}{relationId, tagged id, sequence}, flattened{relationId, id, instance flag, sequence}, precomputed key{monotonic pageSetId, sequence}+ embeddedm_items(final)The single-word 64-bit
PageSetIdmakes the final key the fastest of all tested variants.Performance
Benchmark:
dp_cache_bench <db> scan <connections> <passes> 2048, all runs interleaved; medians of 5 runs.Each version was measured on a database created by itself from the same schema and data: 500,000 rows, 8,192-byte pages, 4,432 data pages, 3 pointer pages, 61 % fill (verified with
gstat).Single and concurrent connections (v5.0-release vs master vs. patched)
jrd_rel/RelationPages) was per-attachment so its map had no lock and scaled; in 6.0RelationPagesis shared viaMetadataCache, and its per-relation mutex made it 3.0–16.7× slower from 4 to 32 connections.Firebird::HashTable::Entryuses virtualisEqual()andget()methods), the rest comes from the unrelated engine parts. From 8 to 32 connections, shared buffer-cache latch contention on pointer-page fetches grows, so avoiding ~960 pointer-pageCCH_FETCH/CCH_HANDOFFcycles per pass per connection outweighs the single-thread baseline gap and makes the patched build 1.06–1.71× faster than 5.0.Trade-offs
{page set, sequence}pairs per attachment still evicts entries (though the old map was capped at 64 per relation);RelationPagesis reset by one attachment, that attachment immediately frees its own cached slots viainvalidateDPNumbers(tdbb), while other attachments' entries under the retiredm_pageSetIdbecome unreachable and age out via clock replacement.Correctness
PageSetId, updated atomically on construction,reset()andreuse(). RecycledRelationPagesobjects,SET TABLESPACEround-trips, declared LTT re-executions, and recreated tables that reuse a droppedrelationIdorpageSpaceIdalways receive a new key, so entries of a previous incarnation are unreachable in every attachment's cache.reset()andreuse(),RelationPages::invalidateDPNumbers(tdbb)also purges all slots belonging to the retiringPageSetIdfrom the current attachment's cache and returns them tom_free, so temporary table and DDL churn does not leave dead entries occupying capacity.DPM_getandDPM_next(page type and flags,dpg_relation,dpg_sequence,dpg_count > 0). When validation (pageOk) fails,relPages->invalidateDPNumber(tdbb, dpSequence)immediately removes the stale entry (so it does not stay markedused = trueif the pointer-page fallback skips an empty slot) before falling through to the pointer page path.DPM_delete→invalidateDPNumber(tdbb, dpSequence + s)). Other page-freeing paths indpm.eppeither release wholerelation page sets (
DPM_delete_relation_pages→relPages->reset(tdbb)) or release only orphan fragment / blob pages (DPM_release,delete_tail), which are never cached.Verification
make run_tests— 120 + 35 (28 existing + 7 new cache tests) + 8 cases pass.tests/relation_pages_cache/verify.sh, not part of the patch): stress with 15,000 keys to force clock eviction,SET TABLESPACEin both directions across connections, single-attachmentSET TABLESPACEround-trip (PRIMARY → TS → DELETE/UPDATE → PRIMARY) and single-attachmentDROP TABLE+CREATE TABLEreusing the relation ID, GTT instance recycling (20 × 2,000 rows), and two attachments with 2,000 GTT rows each — no cross-instance cache hits.ALTER TABLEscenario (rows 2 → 3 → 4) to confirm temporary base page sets and frame-scoped instances are still resolved correctly.Files
src/jrd/DataPageNumberCache.h— new cache (PageSetId, embeddedm_items, clock eviction, hash,invalidateDPNumber/invalidateDPNumbers)src/jrd/Attachment.h—att_dp_number_cache+ lazy accessorsrc/jrd/Relation.h— atomic monotonicm_pageSetId, accessor and invalidation wrappers,tdbbthreaded intoreset()/free()/reuse(), old 64-entry map removedsrc/jrd/Relation.cpp— passtdbbtofree()andreuse()src/jrd/dpm.epp— passtdbbtogetDPNumber/setDPNumber/invalidateDPNumber/reset(), invalidate on!pageOksrc/dsql/DdlNodes.epp— passtdbbtonewPages->reset(...)src/jrd/tests/DataPageNumberCacheTest.cpp— new unit testsbuilds/win32/msvc15/engine_static.vcxproj{,.filters}— new headerbuilds/win32/msvc15/engine_test.vcxproj{,.filters}— new unit testUnit tests
Seven cases in
DataPageNumberCacheSuite(libEngine14_test), all driving a realRelationPagesagainst a fresh pool:Key identity
KeyIncludesPageSet— differentRelationPagesobjects (different page spaces, two distinct objects created for the sameDB_PAGE_SPACE, and an instance page set) get distinct keys and do not collide; an unknown sequence misses.Identity changes / invalidation
ReuseChangesIdentity—reset(nullptr),reset(nullptr, 99)back to the same page space,reuse(nullptr, 99, 7),reset(nullptr)whileinstanceId != 0, andreuse(nullptr, 99, 7)with identical arguments each make previously cached entries unreachable.SetInstanceIdPreservesIdentity—setInstanceId()(GTT retained across a retaining commit) preserves the page-set key so cached data page numbers stay valid.Eviction policy
CapacityAndClockEviction— inserting 2 ×CAPACITYkeys keeps the most recent 100 and proves the first entry was evicted.ClockEvictsOneAtATime— a full cache plus one insert evicts exactly one entry, not a batch.ClockSecondChance— after the sweep clears all reference bits, a recently read entry survives the next eviction while the following cold entry is replaced.Update & removal
RemoveAndReuse—setDPNumberoverwrites an existing entry in place;invalidateDPNumberdrops a single sequence and returns its slot for reuse;invalidateDPNumbersdrops all sequences of one page set while preserving entries of other page sets.