Add write-pause concurrency in test/format and make disagg step-down fully asynchronous

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Test Format
    • Storage Engines - Foundations
    • 554.296
    • SE Transactions - 2026-07-17, SE Transactions - 2026-07-31, SE Transactions - 2026-08-14
    • 5

      Summary

      During a disaggregated step-down in async mode, the step-down checkpoint runs from the background thread while worker threads are still active. This causes a cache overflow that crashes the ingest table.

      Root Cause

      After the drain completes, worker threads resume issuing new transactions with timestamps above step_down_ts (the ingest path). These generate dirty pages that compete with the concurrent step-down checkpoint, which is simultaneously trying to flush pages at the step_down_ts boundary. Both the checkpoint and the new ingest writes compete for the same cache, causing it to overflow.

      Proposed Fix: Pause Worker Writes During Checkpoint

      g.stepdown_ts is already set to a non-zero value (via WT_RELEASE_WRITE_WITH_BARRIER) from when the async thread captures step_down_ts until disagg_switch_roles clears it after the role transition. This signal is visible to all worker threads and can be used to gate write operations.

      Workers should check g.stepdown_ts != WT_TS_NONE before issuing write operations. If the flag is set, the worker skips writes and runs read-only for that operation cycle.

      Effect:

      • No new dirty pages are generated while the checkpoint is running.
      • The checkpoint sees only the bounded set of pages at the step_down_ts boundary — no cache pressure.
      • Worker threads remain live throughout (track_ops continues ticking, read paths are exercised).
      • The checkpoint can continue running from the background thread (no structural reorganization needed).
      • The write pause naturally extends through the post-stepdown follower window (DISAGG_SWITCH_FOLLOWER_OPS_SEC), which is acceptable since we are transitioning to follower anyway and ingest writes past step_down_ts would be invalid in that role.

        1. async-stepdown-flow.svg
          6 kB
        2. async-stepdown-flow.png
          async-stepdown-flow.png
          82 kB

            Assignee:
            Ravi Giri
            Reporter:
            Ravi Giri
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: