Issue Summary

Verify compares a data store checkpoint against the live history store rather than against the history store as of the checkpoint it is verifying. The two views can differ arbitrarily, so verify's history store checks are made against content that was never part of the checkpoint under examination.

Reproducer attached: test_verify_hs_ckpt.py. It fails deterministically on develop, through the public API only:

AssertionError: 0 != 1501 : verify of an unchanged checkpoint walked a different
history store: 0 globally visible records skipped the first time, 1501 the second

cd build && python3 ../test/suite/run.py test_verify_hs_ckpt

How the skew arises

1. WT_SESSION::verify returns EBUSY on a dirty tree. It does not force a checkpoint first, so the tree must already be clean before verify will run.
2. A checkpoint skips clean trees. Once a table is clean its checkpoint list stops advancing, while the history store keeps moving under other tables' activity and under an advancing oldest timestamp.
3. Verify then compares the pinned data store checkpoint against a history store that has moved on.

The reproducer verifies the same WiredTigerCheckpoint.1 twice. Between the two runs the table is never touched, and the test asserts its checkpoint list is unchanged, so the data store side is byte-identical. The cursor_next_hs_tombstone and cursor_prev_hs_tombstone statistics then show verify walking a different history store the second time.

Both counters move, so this covers the next walk in _wt_hs_verify_one() as well as the prev walk in _verify_key_hs().

Why WT-18330 does not close this

WT-18330 fixed the disaggregated follower exposure. Both _wt_hs_verify_one() and verify_key_hs() now pass btree->hs_checkpoint_name to wt_curhs_open(), and the dead session->hs_checkpoint mechanism in _verify_key_hs() was removed along with the unused vs->hs_checkpoint_name.

btree->hs_checkpoint_name is only ever set by the disaggregated uri/<checkpoint> open path (__btree_pin_hs_dhandle(), src/btree/bt_handle.c:55). On a leader and in non-disaggregated WiredTiger it is NULL, so the history store is still opened live and WT-18330 is a no-op for this gap by construction.

Proposed Solution

Verify needs to open the history store as a genuine checkpoint cursor matching the data store checkpoint it is verifying. This is not a variation on the WT-18330 change:

• The history store checkpoint corresponding to a given data store checkpoint has to be resolved. The two files can legitimately come from different global checkpoints, precisely because checkpoints skip clean trees.
• That matching logic already exists for checkpoint cursors in __wt_session_get_btree_ckpt() (src/session/session_dhandle.c), which resolves a matching data store and history store pair together with the checkpoint snapshot. Reusing it is the obvious starting point.
• Verify already routes through _wt_session_get_btree_ckpt() (src/schema/schema_worker.c:69), but passes NULL for both the history store dhandle and the checkpoint snapshot, and WT_SESSION::verify has no checkpoint configuration option. That function early-returns to _wt_session_get_dhandle(session, uri, NULL, ...) when no checkpoint is configured, so the matching machinery never runs and WT_READING_CHECKPOINT(session) is false throughout verify.
• Verify also iterates every checkpoint itself via bm->checkpoint_load() in __verify_one_checkpoint() rather than opening a checkpoint dhandle per checkpoint, so there is no checkpoint dhandle to hang a matching history store handle off. History store verification only runs against the last checkpoint, which narrows this but does not remove it.

Note that history store verification only runs against the last checkpoint (vs->skip_hs = skip_hs || !last_ckpt in src/btree/bt_vrfy.c). That narrows the problem but does not remove it: the last checkpoint of a clean tree can be arbitrarily old relative to the history store, which is exactly what the reproducer exploits.

Definition of Done

• test_verify_hs_ckpt.py passes and is committed.
• Verify compares a data store checkpoint against the history store as of that checkpoint.
• Existing verify coverage is unchanged: test_verify3, test_verify_hs_overlap, test_verify_disagg, test_verify_disagg02, test_verify_disagg03, test_verify_disagg04.
• The test asserts the invariant first == second rather than a specific count, because the post-fix values depend on how a checkpoint history store cursor evaluates global visibility. If that invariant needs adjusting for a legitimate reason, the reason is recorded here rather than the assertion simply being relaxed.