Step-down's btree marking loop races the sweep server on stable dhandle flags

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Layered Tables
    • Storage Engines - Foundations
    • 527.916
    • SE Foundations - 2026-09-01, SE Foundations - 2026-09-15
    • None

      __disagg_mark_btrees_readonly_then_step_down() walks the dhandle list under only the handle-list read lock and, for each open disaggregated stable btree, sets WT_BTREE_READONLY on the btree and WT_DHANDLE_OUTDATED on the dhandle. It never acquires the handles it modifies.

      The sweep server can close the same stable btree dhandles concurrently: __sweep_expire_one() takes the dhandle write lock (which the marking loop does not take) and __wt_conn_dhandle_close() read-modify-writes the same dhandle->flags word (F_SET(WT_DHANDLE_DEAD), F_CLR(WT_DHANDLE_OPEN)). Two writers with disjoint lock sets doing plain non-atomic flag updates: either side's bit can be lost. In addition, the loop checks WT_DHANDLE_OPEN but not WT_DHANDLE_DEAD, so it will happily run __wt_evict_file_exclusive_on() and set flags on a btree the sweep server already closed (a dead-but-not-yet-discarded handle needs no narrow timing at all).

      Because of surrounding checks, every reachable outcome is mostly benign today:

      • Lost WT_DHANDLE_DEAD: the handle briefly claims to be open and alive over a closed btree, but the sweep expire path re-processes OUTDATED handles unconditionally, so the next sweep pass re-marks it dead and cleanup completes – the damage is one delayed sweep pass, not a wedged handle. The tree was clean when swept, so eviction walking it in the window does not touch the block manager.
      • Lost WT_DHANDLE_OUTDATED: moot, the handle is dead and will be discarded, which is all the flag was for.
      • Flag races on btree->flags (WT_BTREE_READONLY vs WT_BTREE_CLOSED): losing WT_BTREE_CLOSED only causes a second close, which is idempotent.
      • No use-after-free is possible: discarding the handle requires the handle-list write lock, which the loop's read lock blocks.

      Still worth fixing: these are genuine data races (undefined behavior, TSan-reportable), and the benignity depends on incidental properties of the current close/expire code rather than any stated invariant. Suggested fix: take the dhandle write lock per handle in the marking loop (rare path, cost irrelevant) and skip WT_DHANDLE_DEAD handles.

      __wti_conn_dhandle_outdated() has the same unlocked F_SET(WT_DHANDLE_OUTDATED) pattern; that callsite is production-reachable (follower checkpoint pickup) and is handled separately with higher priority in WT-18452 (AF-19791).

      Supersedes WT-18326, whose scenarios wrongly assumed the sweep server closes layered dhandles.

      Note: as of dsc-release-5 (e8373f57) the marking loop itself is unreachable from mongod - no caller of demoteToFollower() and every server step-down exits the process (SLS-1414). The races here become reachable once in-process step-down (WT-17091 / featureFlagElegantStepDownForDisagg) is wired up.

            Assignee:
            Ivan Kochin
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: