Disagg: follower's live shared HS dhandle is retired only by conditional, asynchronous pickup-time outdating

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: DHandles
    • None
    • Storage Engines
    • 70.238
    • None
    • None

      What we found

      • _wt_hs_open() creates the shared history store at connection open on every node (hs_conn.c), which implicitly opens the live file:WiredTigerSharedHS.wt_stable dhandle. Unlike the shared metadata table – which is expired immediately on followers (_disagg_metadata_table_init, conn_layered.c, see FIXME-WT-17040) – nothing expires the live shared HS handle on a follower.
      • Verified empirically with an assert probe at checkpoint pickup: on a follower at pickup time, __wt_conn_dhandle_find(WT_HS_URI_SHARED, NULL) finds the live HS dhandle open (open=1 inuse=0), e.g. in test_layered_stepup05.
      • The only mechanism that retires it is the unconditional outdating call at the end of __disagg_update_file_meta() (conn_layered_checkpoint_pick_up.c, the FIXME-WT-17772 call). That call is:
        • conditional – skipped when the HS checkpoint is unchanged in that pickup (early return when the local checkpoint already matches the shared one), and
        • asynchronousF_SET(WT_DHANDLE_OUTDATED) only affects future opens; in-flight references keep using the old btree until they drain.

      Why checkpoint readers are safe (for contrast)

      Stable checkpoint opens pair the HS checkpoint atomically: _wt_btree_open takes the checkpoint lock around _btree_pin_hs_dhandle_and_get_meta_checkpoint (bt_handle.c), pickup holds the same lock for its entire run, and a superseded checkpoint fails the open with EBUSY so the layered cursor retries. Checkpoint-view reads therefore cannot mix a new stable checkpoint with an old HS checkpoint.

      The exposure

      The live HS handle has no such pairing:

      1. Internal paths open the live HS with a NULL checkpoint (e.g. __wt_hs_btree_truncate, hs_cursor.c). Any such access on a follower that races a pickup keeps operating on the pre-pickup HS btree after the local metadata (and every paired checkpoint reader) has moved to the new checkpoint. The OUTDATED mark does not fence in-flight users.
      2. Step-up depends on this chain transitively: the forced pickup runs before drain, and the new leader's HS accesses reuse any open, non-outdated live HS dhandle without reopening (__wt_session_get_dhandle reuses OPEN handles). Today the pickup-time mark is the only thing preventing the follower-era live HS handle (opened at connection start) from surviving into the leader era with a stale root. Removing or moving the pickup-time outdating (WT-17772; the WT-18452 race also lives on this call site) silently breaks this hidden dependency.

      Status of evidence

      Demonstrated: the open, unretired live shared HS dhandle on a follower at pickup time, and the fact that pickup-time outdating is its sole retirement mechanism. Not demonstrated: an end-to-end stale read/write through the live handle (no live-HS user was caught in the act on a follower in the tests run so far).

      Suggested fix

      • Expire the live shared HS dhandle on followers right after creation, exactly as __disagg_metadata_table_init does for the shared metadata table, and/or force-close live handles at role change (WT-17772).
      • Add an assertion (in the spirit of FIXME-WT-18357) that a follower never holds an open, non-outdated live shared HS handle.
      • Longer term this is subsumed by WT-18356 (do not keep the live handle at all); this ticket records the concrete dependency and race window so the pickup-time outdating is not removed before the handle's lifecycle is fixed.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: