-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Engines - Foundations
-
2.645
-
None
-
None
Follow-up to WT-18015 (#14416), which audited WT_BTREE flags and moved WT_BTREE_READONLY/WT_BTREE_SKIP_CKPT into the atomic flags_atomic word because they are written on a live, already-open handle while other threads plain-read the same word.
That audit did not cover the dhandle and layered-table flag words touched in the very same step-down loop.
Finding 1 (primary) — WT_DATA_HANDLE.flags
src/include/dhandle.h:167-171 documents: "These flags can only be changed when a dhandle is locked exclusively." There is no flags_atomic word on WT_DATA_HANDLE at all (unlike WT_BTREE).
Two write sites violate this for WT_DHANDLE_OUTDATED:
- src/conn/conn_layered.c:1497 F_SET(dhandle, WT_DHANDLE_OUTDATED) — the very next statement after the F_SET_ATOMIC_32(btree, WT_BTREE_READONLY) that
WT-18015fixed. Holds only the handle-list read lock + evict-file-exclusive on that one btree; not dhandle-exclusive. - src/conn/conn_dhandle.c:352 (__wti_conn_dhandle_outdated) — the handle-list read lock is released before the F_SET; only a refcount is held.
In non-TSAN builds F_SET is a plain |= (src/include/misc.h:225), and all 12 dhandle flags share one uint16_t. So this is not merely a stale/torn read but a lost update: F_SET(dhandle, OUTDATED) racing with F_SET(dhandle, WT_DHANDLE_DEAD) (conn_dhandle.c:528) or F_CLR(dhandle, WT_DHANDLE_OPEN) (conn_dhandle.c:544) can silently drop either bit. No reader (eviction hot path in btree_inline.h:48, WT_DHANDLE_CAN_REOPEN in dhandle.h:64-67, sweep, checkpoint, __wt_conn_dhandle_find) shares a mutual-exclusion lock with either writer.
Finding 2 — WT_LAYERED_TABLE.flags
src/conn/conn_layered.c:1471 clears WT_LAYERED_TABLE_STEP_DOWN_CREATED on every open layered dhandle during step-down, holding only the handle-list read lock (no evict-exclusive at all here). Concurrent lock-free readers on the cursor hot path: src/cursor/cur_layered.c:431, :1250, src/cursor/cur_stat.c:440. WT_LAYERED_TABLE.flags is a plain uint8_t with no atomic word.
Finding 3 (lower priority, noted for awareness)
WT_DISAGG_STABLE_TOMBSTONE_ENCODING is written at checkpoint pickup on a live connection (conn_layered_checkpoint_pick_up.c:2013,2015) and plain-read by cursor threads (cur_layered.c:84); WT_CONNECTION_IMPL.disaggregated_storage.flags is a plain uint8_t. Mitigated somewhat by the panic-on-change-after-adoption check, so the real race window is only first adoption.
Proposed fix
Follow the WT-18015 precedent:
- Add a flags_atomic word to WT_DATA_HANDLE and move WT_DHANDLE_OUTDATED into it, converting all read/write sites to F_ATOMIC. Care needed: dhandle.h:167 reserves values over 0xfff for WT_BTREE_* so the two flag sets can be combined in one call — verify no site passes OUTDATED combined with a btree mask. WT_DHANDLE_CAN_REOPEN tests OUTDATED together with DEAD/DROPPED/OPEN in one F_MASK and must be split into a non-atomic mask plus a separate atomic test, preserving semantics — this is exactly the kind of split that produced the || vs && bug fixed in WT-18534.
- Make WT_LAYERED_TABLE.flags atomic (or otherwise synchronize WT_LAYERED_TABLE_STEP_DOWN_CREATED), since it's a small field with only two flags.
Verification
- sf + wb.
- Grep every converted flag to confirm no leftover plain F_ISSET(dhandle, WT_DHANDLE_OUTDATED) site was missed (a miss compiles but silently reads the wrong word).
- Run the disagg/layered step-down suite locally.
- Submit an Evergreen patch including a TSAN variant — FLD_SET only becomes atomic under TSAN_BUILD (misc.h:210-221), so that's the variant that actually proves the fix.