Step-down's btree marking loop races the sweep server on stable dhandle flags

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 4.781
    • None
    • None

      __disagg_mark_btrees_readonly_then_step_down() walks the dhandle list under only the handle-list read lock and, for each open disaggregated stable btree, sets WT_BTREE_READONLY on the btree and WT_DHANDLE_OUTDATED on the dhandle. It never acquires the handles it modifies.

      The sweep server can close the same stable btree dhandles concurrently: _sweep_expire_one() takes the dhandle write lock (which the marking loop does not take) and wt_conn_dhandle_close() read-modify-writes the same dhandle->flags word (F_SET(WT_DHANDLE_DEAD), F_CLR(WT_DHANDLE_OPEN)). Two writers with disjoint lock sets doing plain non-atomic flag updates: either side's bit can be lost. In addition, the loop checks WT_DHANDLE_OPEN but not WT_DHANDLE_DEAD, so it will happily run _wt_evict_file_exclusive_on() and set flags on a btree the sweep server already closed (a dead-but-not-yet-discarded handle needs no narrow timing at all).

      Because of surrounding checks, every reachable outcome is mostly benign today:

      • Lost WT_DHANDLE_DEAD: the handle briefly claims to be open and alive over a closed btree, but the sweep expire path re-processes OUTDATED handles unconditionally, so the next sweep pass re-marks it dead and cleanup completes – the damage is one delayed sweep pass, not a wedged handle. The tree was clean when swept, so eviction walking it in the window does not touch the block manager.
      • Lost WT_DHANDLE_OUTDATED: moot, the handle is dead and will be discarded, which is all the flag was for.
      • Flag races on btree->flags (WT_BTREE_READONLY vs WT_BTREE_CLOSED): losing WT_BTREE_CLOSED only causes a second close, which is idempotent.
      • No use-after-free is possible: discarding the handle requires the handle-list write lock, which the loop's read lock blocks.

      Still worth fixing: these are genuine data races (undefined behavior, TSan-reportable), and the benignity depends on incidental properties of the current close/expire code rather than any stated invariant. Suggested fix: take the dhandle write lock per handle in the marking loop (rare path, cost irrelevant) and skip WT_DHANDLE_DEAD handles.

      __wti_conn_dhandle_outdated() has the same unlocked F_SET(WT_DHANDLE_OUTDATED) pattern; handle separately or note as follow-up.

      Supersedes WT-18326, whose scenarios wrongly assumed the sweep server closes layered dhandles.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: