test/format: advance timestamps before reconfiguring to leader to avoid cache-stuck on step-up

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • WT12.0.0, 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Test Format
    • None
    • Storage Engines - Foundations
    • 292.889
    • None
    • None

      Summary

      test/format hits a cache-stuck hang across a disaggregated step-up. Root cause is the ordering in format_disagg.c: we reconfigure to leader before advancing timestamps.

      Root cause

      In disagg_switch_role() (test/format/format_disagg.c, the follower -> leader branch), we do:

      testutil\_check\(g.wts\_conn\->reconfigure\(g.wts\_conn, "disaggregated=\(role=leader\)"\)\);
      
      /\* Advance timestamps to cover all in\-memory commits from the follower phase. \*/
      timestamp\_sync\_threads\_commit\_ts\(\);
      timestamp\_once\(session, false, false\);
      

      While in the follower era, moving stable forward does not let us evict ingest content -- ingest is pruned based on the last checkpoint timestamp (the last checkpoint picked up). But once we step up with stable already moved all the way forward, eviction is allowed to run on the stable btrees and can evict those pages.

      Because the reconfigure to leader happens first, at step-up time stable is still pinned at stepdown_ts, so eviction cannot make progress on the pages drained/handed over from the follower phase, and the cache fills up and gets stuck.

      This did not reproduce before the async step-down work because as a follower we relied on the timestamp threads to keep pushing stable forward. With that change, stable stays pinned at stepdown_ts for the whole follower era, so nothing advances it before step-up.

      Fix

      Move the timestamp advance ahead of the role reconfigure:

      timestamp\_sync\_threads\_commit\_ts\(\);
      timestamp\_once\(session, false, false\);
      
      testutil\_check\(g.wts\_conn\->reconfigure\(g.wts\_conn, "disaggregated=\(role=leader\)"\)\);
      

      Test-only change, confined to test/format.

            Assignee:
            Sid Mahajan
            Reporter:
            Sid Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: