-
Type:
Bug
-
Resolution: Works as Designed
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: DHandles
-
None
-
Storage Engines
-
70.238
-
None
-
None
What we found
- _wt_hs_open() creates the shared history store at connection open on every node (hs_conn.c), which implicitly opens the live file:WiredTigerSharedHS.wt_stable dhandle. Unlike the shared metadata table – which is expired immediately on followers (_disagg_metadata_table_init, conn_layered.c, see FIXME-WT-17040) – nothing expires the live shared HS handle on a follower.
- Verified empirically with an assert probe at checkpoint pickup: on a follower at pickup time, __wt_conn_dhandle_find(WT_HS_URI_SHARED, NULL) finds the live HS dhandle open (open=1 inuse=0), e.g. in test_layered_stepup05.
- The only mechanism that retires it is the unconditional outdating call at the end of __disagg_update_file_meta() (conn_layered_checkpoint_pick_up.c, the FIXME-WT-17772 call). That call is:
- conditional – skipped when the HS checkpoint is unchanged in that pickup (early return when the local checkpoint already matches the shared one), and
- asynchronous – F_SET(WT_DHANDLE_OUTDATED) only affects future opens; in-flight references keep using the old btree until they drain.
Why checkpoint readers are safe (for contrast)
Stable checkpoint opens pair the HS checkpoint atomically: _wt_btree_open takes the checkpoint lock around _btree_pin_hs_dhandle_and_get_meta_checkpoint (bt_handle.c), pickup holds the same lock for its entire run, and a superseded checkpoint fails the open with EBUSY so the layered cursor retries. Checkpoint-view reads therefore cannot mix a new stable checkpoint with an old HS checkpoint.
The exposure
The live HS handle has no such pairing:
- Internal paths open the live HS with a NULL checkpoint (e.g. __wt_hs_btree_truncate, hs_cursor.c). Any such access on a follower that races a pickup keeps operating on the pre-pickup HS btree after the local metadata (and every paired checkpoint reader) has moved to the new checkpoint. The OUTDATED mark does not fence in-flight users.
- Step-up depends on this chain transitively: the forced pickup runs before drain, and the new leader's HS accesses reuse any open, non-outdated live HS dhandle without reopening (__wt_session_get_dhandle reuses OPEN handles). Today the pickup-time mark is the only thing preventing the follower-era live HS handle (opened at connection start) from surviving into the leader era with a stale root. Removing or moving the pickup-time outdating (WT-17772; the WT-18452 race also lives on this call site) silently breaks this hidden dependency.
Status of evidence
Demonstrated: the open, unretired live shared HS dhandle on a follower at pickup time, and the fact that pickup-time outdating is its sole retirement mechanism. Not demonstrated: an end-to-end stale read/write through the live handle (no live-HS user was caught in the act on a follower in the tests run so far).
Suggested fix
- Expire the live shared HS dhandle on followers right after creation, exactly as __disagg_metadata_table_init does for the shared metadata table, and/or force-close live handles at role change (WT-17772).
- Add an assertion (in the spirit of FIXME-WT-18357) that a follower never holds an open, non-outdated live shared HS handle.
- Longer term this is subsumed by WT-18356 (do not keep the live handle at all); this ticket records the concrete dependency and race window so the pickup-time outdating is not removed before the handle's lifecycle is fixed.
- is related to
-
WT-18452 Unlocked WT_DHANDLE_OUTDATED flag update in __wti_conn_dhandle_outdated races dhandle close
-
- In Code Review
-
-
WT-17040 Investigate whether the creation of shared metadata is necessary on followers
-
- Open
-
-
WT-17772 Consider marking data handles outdated at step-down or step-up
-
- Open
-
-
WT-18356 Disagg: a follower should not keep a live shared history store handle
-
- Backlog
-
-
WT-18357 Disagg: assert a follower holds no writable live stable handle and narrow the open refusal to the step-down race
-
- Backlog
-
- related to
-
WT-17772 Consider marking data handles outdated at step-down or step-up
-
- Open
-
-
WT-18356 Disagg: a follower should not keep a live shared history store handle
-
- Backlog
-