Layered cursor stable-mode assert races with disaggregated role step-down

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cursors

      Summary

      __clayered_assert_stable_mode() in src/cursor/cur_layered.c asserts that the stable cursor’s btree is read-write for a leader and read-only for a follower. This invariant holds in steady state but is racy during step-down, so the assertion can fire spuriously on diagnostic builds.

      Assertion

       
      
      {{/* The stable cursor's btree must be read-write for a leader and read-only for a follower. */WT_ASSERT(CUR2S(clayered),
      (clayered->last_role == WTI_CLAYERED_ROLE_LEADER) !=
      F_ISSET(CUR2BT(clayered->stable_cursor), WT_BTREE_READONLY));}}
      
      

      Race description

      _disagg_mark_btrees_readonly_then_step_down() (in src/conn/conn_layered.c) first marks live disaggregated btrees WT_BTREE_READONLY, and only afterwards flips conn->layered_table_manager.leader = false. A layered cursor samples the leader flag locklessly in _clayered_enter(); if it samples between those two steps it still sees role == LEADER while its live stable btree is already read-only, so the assertion fails. The follower direction is safe: followers always open the stable table at a checkpoint URI, and checkpoint dhandles are unconditionally read-only.

      How it was found

      Exposed by the async step-down work in test/format (WT-18044):

      • Worker threads continue read operations while a background thread runs reconfigure("disaggregated=(role=follower)").
      • Layered cursors can therefore enter during the step-down window.
      • The previous synchronous path joined all workers before reconfiguring and never exercised this window.

      Impact

      • No correctness issue: no data is read or written incorrectly.
      • The window only allows a leader-role cursor to observe its (already write-quiesced) live stable btree as read-only.
      • The actual read path behaves correctly, and once the role change is visible the cursor will reopen the stable table on the next operation.
      • Symptom: a spurious WT_ASSERT abort on diagnostic builds during step-down.

            Assignee:
            Sid Mahajan
            Reporter:
            Ravi Giri
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: