-
Type:
Bug
-
Resolution: Works as Designed
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
None
-
Storage Engines
-
30.093
-
None
-
None
__disagg_mark_btrees_readonly_then_step_down() walks the connection's dhandle list under only the handle-list read lock and updates per-handle flags directly: F_CLR(WT_LAYERED_TABLE_STEP_DOWN_CREATED) on layered->flags, F_SET(WT_DHANDLE_OUTDATED) on dhandle->flags, F_SET(WT_BTREE_READONLY) on btree->flags. It never acquires the dhandles it modifies.
Handle close paths — _sweep_expire_one() on the sweep server and a session's release-with-discard through wt_session_release_dhandle() — hold the dhandle write lock but not the schema lock, so they can run concurrently with the loop and read-modify-write the same flag words: F_CLR(WT_LAYERED_TABLE_OPEN) in wt_schema_close_layered() and the WT_DHANDLE_DEAD/WT_DHANDLE_OPEN updates in _wt_conn_dhandle_close(). Two writers, disjoint lock sets, plain non-atomic read-modify-writes on shared bytes: either side's update can be lost. No weak memory required — this is a plain interleaving, reachable on x86.
The lost update is persisted into the dhandle struct, which stays on the connection list across close for reopen, so correctly synchronized code consumes the corruption much later:
- lost WT_LAYERED_TABLE_OPEN clear → the close-idempotence guard in __wt_schema_close_layered() is defeated → the next close re-frees key_format/value_format/ingest_uri/stable_uri → double free;
- lost WT_LAYERED_TABLE_STEP_DOWN_CREATED clear → the flag's only cleaner is this loop and reopen never clears it → after a later step-up, cursor operations on the reopened handle skip the stable constituent → silent ingest-only reads on a leader;
- lost WT_DHANDLE_OUTDATED set → a later step-up reuses the handle's resident pages instead of opening fresh — exactly the unresolved-prepared-cell problem the flag exists to prevent;
- lost WT_DHANDLE_DEAD/WT_DHANDLE_OPEN updates → sweep and handle-lookup bookkeeping desync from the closed btree underneath.
All payloads are silent and fire long after the race, in unrelated stacks. Note the read side is fine: cursor operations never race the close (exclusivity covers them), and readers of these flags either tolerate staleness or are ordered by the role publish — the defect is strictly writer-vs-writer.
A possible direction: take the dhandle write lock per handle in the marking loop (it is a rare path, so the cost is irrelevant); independently, clearing a stale WT_LAYERED_TABLE_STEP_DOWN_CREATED at dhandle open when the derive-condition doesn't hold would neutralize the resurrection payload.
Example interleavings for both directions are in the comments.
- is related to
-
WT-16982 Provide long term solution layered dhandles and ingest tables
-
- Backlog
-