Verify opens the history store live instead of at the checkpoint pinned by the stable btree

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: History Store, Verify
    • None
    • Storage Engines, Storage Engines - Persistence
    • 26.826
    • SE Persistence backlog
    • None

      Issue Summary

      __wt_hs_verify_one() in src/history/hs_verify.c opens its history store cursor with a NULL checkpoint name:

      WT_ERR(__wt_curhs_open(session, btree_id, NULL, NULL, &hs_cursor));
      

      On a disaggregated follower this opens a live, writable handle on the shared history store (file:WiredTigerSharedHS.wt_stable), even though the data store side of the comparison is a read-only handle opened from one specific checkpoint. The correct argument is btree->hs_checkpoint_name: the shared history store checkpoint that was pinned when the stable btree handle was opened.

      This is the same defect [WT-17358] fixed in the sibling function _hs_verify() a few lines below in the same file. wt_hs_verify_one() was missed. _wt_hs_search() already passes the pinned name correctly (src/history/hs_cursor.c:118).

      Context

      How a follower reaches this call site with a pinned checkpoint name available:

      • WT_SESSION::verify() on a layered table routes through __schema_layered_stable_worker_verify() (src/schema/schema_worker.c:160), which on a follower opens the stable constituent as file:X.wt_stable/<checkpoint>.
      • That URI form makes _wt_btree_open() call _btree_pin_hs_dhandle() (src/btree/bt_handle.c:55), which pins the matching shared history store checkpoint into btree->hs_checkpoint_name and sets WT_BTREE_READONLY.
      • _wt_verify() then calls _wt_hs_verify_one(session, btree->id) for the last checkpoint (src/btree/bt_vrfy.c:479), which discards that pinned name.

      Non-disaggregated builds are unaffected. btree->hs_checkpoint_name is only ever set by the uri/<checkpoint> open path, so it is NULL on a leader and in local WiredTiger, and the fix below is a no-op there.

      How to prove the call site is wrong

      Temporarily add this at the top of __wt_hs_verify_one():

      WT_ASSERT_ALWAYS(session, S2BT(session)->hs_checkpoint_name == NULL,
        "INSTRUMENT: hs_verify_one with pinned HS checkpoint %s, readonly %d, disagg %d",
        S2BT(session)->hs_checkpoint_name, F_ISSET(S2BT(session), WT_BTREE_READONLY) ? 1 : 0,
        F_ISSET(S2BT(session), WT_BTREE_DISAGGREGATED) ? 1 : 0);
      

      Then run a disaggregated test that calls WT_SESSION::verify() against a populated layered table on a follower (see the proposed test/suite/test_verify_disagg04.py below). The assertion fires:

      [1786514983:761971][45846:0x1f643e180], file:test_verify_disagg04.wt_stable/WiredTigerCheckpoint.3,
      WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: int __wt_hs_verify_one(WT_SESSION_IMPL *, uint32_t), 117:
      WiredTiger assertion failed:
      '((WT_BTREE *)(session)->dhandle->handle)->hs_checkpoint_name == ((void*)0)'.
      INSTRUMENT: hs_verify_one with pinned HS checkpoint WiredTigerCheckpoint.2, readonly 1, disagg 1
      

      The data store is being verified at WiredTigerCheckpoint.3 while a pinned shared history store checkpoint WiredTigerCheckpoint.2 is available and ignored.

      Why this does not fail today

      Several attempts to turn this into a failing test did not succeed on current develop. It is a latent defect, not an active one:

      • The assertion [WT-17358] hit (_wt_page_modify_set() asserting !DISAGG || leader) fired because _inmem_row_leaf() instantiated tombstones for stop-timestamp records. WT-17601 (commit e4e9a03db7) removed that. Instantiation on page read is now gated on WT_TIME_WINDOW_HAS_PREPARE only, and prepared updates reach the data store rather than the history store, so a live history store handle no longer dirties pages on read.
      • Content skew between the live history store and the pinned checkpoint cannot open up either. Verify holds the checkpoint lock (src/session/session_api.c:1857), checkpoint pickup takes the same lock (src/conn/conn_layered_checkpoint_pick_up.c:1949), and pickup marks both the uri/<checkpoint> and the live shared dhandles outdated (same file, lines 718 and 729).

      The exposure reopens the moment update instantiation on page read is widened again, or a live shared handle on a follower is dirtied by any other route. Both sibling call sites already guard against this, so the inconsistency is worth removing on its own.

      Proposed Solution

      In __wt_hs_verify_one():

      btree = S2BT(session);
      
      if (WT_URI_IS_STABLE_CHECKPOINT(session->dhandle->name) && btree->hs_checkpoint_name == NULL)
          return (0);
      
      WT_ERR(__wt_curhs_open(session, btree_id, btree->hs_checkpoint_name, NULL, &hs_cursor));
      

      Key the early return off WT_URI_IS_STABLE_CHECKPOINT rather than the DISAGGREGATED && READONLY condition that _wt_hs_search() uses. Verify sets WT_BTREE_READONLY on every handle it opens, so the _wt_hs_search() condition would silently skip history store verification on the leader.

      Definition of Done

      • __wt_hs_verify_one() reads the history store at the checkpoint pinned by the stable btree rather than through a live handle.
      • A new test/suite/test_verify_disagg04.py drives WT_SESSION::verify() against a populated shared history store on a follower, giving the path regression coverage. Write data at ts=5, overwrite at ts=10, delete every other key at ts=15, checkpoint after each stage, then advance a follower and verify from it.
      • test_verify_disagg, test_verify_disagg02, test_verify_disagg03 and the new test pass, along with the verify, history store, disaggregated and layered Python suites.
      • The PR description states plainly that the new test is path coverage, not a failing-before / passing-after reproducer, so a future reader does not assume the bug was reproducible.

        1. test_verify_hs_ckpt.py
          6 kB
          Etienne Petrel

            Assignee:
            Etienne Petrel
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: