-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Test Format
-
None
-
Storage Engines - Foundations
-
292.889
-
None
-
None
Summary
test/format hits a cache-stuck hang across a disaggregated step-up. Root cause is the ordering in format_disagg.c: we reconfigure to leader before advancing timestamps.
Root cause
In disagg_switch_role() (test/format/format_disagg.c, the follower -> leader branch), we do:
testutil\_check\(g.wts\_conn\->reconfigure\(g.wts\_conn, "disaggregated=\(role=leader\)"\)\); /\* Advance timestamps to cover all in\-memory commits from the follower phase. \*/ timestamp\_sync\_threads\_commit\_ts\(\); timestamp\_once\(session, false, false\);
While in the follower era, moving stable forward does not let us evict ingest content -- ingest is pruned based on the last checkpoint timestamp (the last checkpoint picked up). But once we step up with stable already moved all the way forward, eviction is allowed to run on the stable btrees and can evict those pages.
Because the reconfigure to leader happens first, at step-up time stable is still pinned at stepdown_ts, so eviction cannot make progress on the pages drained/handed over from the follower phase, and the cache fills up and gets stuck.
This did not reproduce before the async step-down work because as a follower we relied on the timestamp threads to keep pushing stable forward. With that change, stable stays pinned at stepdown_ts for the whole follower era, so nothing advances it before step-up.
Fix
Move the timestamp advance ahead of the role reconfigure:
timestamp\_sync\_threads\_commit\_ts\(\); timestamp\_once\(session, false, false\); testutil\_check\(g.wts\_conn\->reconfigure\(g.wts\_conn, "disaggregated=\(role=leader\)"\)\);
Test-only change, confined to test/format.