-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
Storage Engines - Foundations
-
527.916
-
SE Foundations - 2026-09-01, SE Foundations - 2026-09-15
-
None
__disagg_mark_btrees_readonly_then_step_down() walks the dhandle list under only the handle-list read lock and, for each open disaggregated stable btree, sets WT_BTREE_READONLY on the btree and WT_DHANDLE_OUTDATED on the dhandle. It never acquires the handles it modifies.
The sweep server can close the same stable btree dhandles concurrently: __sweep_expire_one() takes the dhandle write lock (which the marking loop does not take) and __wt_conn_dhandle_close() read-modify-writes the same dhandle->flags word (F_SET(WT_DHANDLE_DEAD), F_CLR(WT_DHANDLE_OPEN)). Two writers with disjoint lock sets doing plain non-atomic flag updates: either side's bit can be lost. In addition, the loop checks WT_DHANDLE_OPEN but not WT_DHANDLE_DEAD, so it will happily run __wt_evict_file_exclusive_on() and set flags on a btree the sweep server already closed (a dead-but-not-yet-discarded handle needs no narrow timing at all).
Because of surrounding checks, every reachable outcome is mostly benign today:
- Lost WT_DHANDLE_DEAD: the handle briefly claims to be open and alive over a closed btree, but the sweep expire path re-processes OUTDATED handles unconditionally, so the next sweep pass re-marks it dead and cleanup completes – the damage is one delayed sweep pass, not a wedged handle. The tree was clean when swept, so eviction walking it in the window does not touch the block manager.
- Lost WT_DHANDLE_OUTDATED: moot, the handle is dead and will be discarded, which is all the flag was for.
- Flag races on btree->flags (WT_BTREE_READONLY vs WT_BTREE_CLOSED): losing WT_BTREE_CLOSED only causes a second close, which is idempotent.
- No use-after-free is possible: discarding the handle requires the handle-list write lock, which the loop's read lock blocks.
Still worth fixing: these are genuine data races (undefined behavior, TSan-reportable), and the benignity depends on incidental properties of the current close/expire code rather than any stated invariant. Suggested fix: take the dhandle write lock per handle in the marking loop (rare path, cost irrelevant) and skip WT_DHANDLE_DEAD handles.
__wti_conn_dhandle_outdated() has the same unlocked F_SET(WT_DHANDLE_OUTDATED) pattern; that callsite is production-reachable (follower checkpoint pickup) and is handled separately with higher priority in WT-18452 (AF-19791).
Supersedes WT-18326, whose scenarios wrongly assumed the sweep server closes layered dhandles.
Note: as of dsc-release-5 (e8373f57) the marking loop itself is unreachable from mongod - no caller of demoteToFollower() and every server step-down exits the process (SLS-1414). The races here become reachable once in-process step-down (WT-17091 / featureFlagElegantStepDownForDisagg) is wired up.
- is blocked by
-
WT-18452 Unlocked WT_DHANDLE_OUTDATED flag update in __wti_conn_dhandle_outdated races dhandle close
-
- In Code Review
-
- is related to
-
WT-18452 Unlocked WT_DHANDLE_OUTDATED flag update in __wti_conn_dhandle_outdated races dhandle close
-
- In Code Review
-
-
WT-18326 Step-down's dhandle marking loop updates flag words without the dhandle lock, racing handle close
-
- Closed
-
-
WT-17091 Investigate and implement step-down for publish
-
- Closed
-
-
WT-18542 Audit and document the WT_DHANDLE_OUTDATED lifecycle and its coupling with dhandle->session_inuse
-
- Needs Scheduling
-
- related to
-
WT-18325 Step-down's dhandle marking loop updates flag words without the dhandle lock, racing handle close
-
- Closed
-
-
WT-18533 Read-only cursor scan can dirty a live stable btree across step-down - "disaggregated btree dirtied while not leader" assertion
-
- Needs Scheduling
-
-
WT-18259 Investigate reads racing an async step-down against the stable table being marked read-only and outdated
-
- Closed
-
-
WT-18542 Audit and document the WT_DHANDLE_OUTDATED lifecycle and its coupling with dhandle->session_inuse
-
- Needs Scheduling
-