-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Test Format
-
Storage Engines - Foundations
-
554.303
-
SE Transactions - 2026-07-17, SE Transactions - 2026-07-31, SE Transactions - 2026-08-14
-
5
Summary
During a disaggregated step-down in async mode, the step-down checkpoint runs from the background thread while worker threads are still active. This causes a cache overflow that crashes the ingest table.
—
Root Cause
After the drain completes, worker threads resume issuing new transactions with timestamps above step_down_ts (the ingest path). These generate dirty pages that compete with the concurrent step-down checkpoint, which is simultaneously trying to flush pages at the step_down_ts boundary. Both the checkpoint and the new ingest writes compete for the same cache, causing it to overflow.
—
Proposed Fix: Pause Worker Writes During Checkpoint
g.stepdown_ts is already set to a non-zero value (via WT_RELEASE_WRITE_WITH_BARRIER) from when the async thread captures step_down_ts until disagg_switch_roles clears it after the role transition. This signal is visible to all worker threads and can be used to gate write operations.
Workers should check g.stepdown_ts != WT_TS_NONE before issuing write operations. If the flag is set, the worker skips writes and runs read-only for that operation cycle.
Effect:
- No new dirty pages are generated while the checkpoint is running.
- The checkpoint sees only the bounded set of pages at the step_down_ts boundary — no cache pressure.
- Worker threads remain live throughout (track_ops continues ticking, read paths are exercised).
- The checkpoint can continue running from the background thread (no structural reorganization needed).
- The write pause naturally extends through the post-stepdown follower window (DISAGG_SWITCH_FOLLOWER_OPS_SEC), which is acceptable since we are transitioning to follower anyway and ingest writes past step_down_ts would be invalid in that role.
- related to
-
WT-18179 Layered cursor stable-mode assert races with disaggregated role step-down
-
- In Progress
-
-
WT-18045 Add debugging and code-cleanup to elegant step-down implementation in test/format
-
- Open
-
-
WT-18221 Decide the destiny of the async step-down tests and formalize the coverage proof
-
- Backlog
-
-
WT-17895 Core Async Stepdown Implementation
-
- Closed
-