Issue Summary

WT_SESSION::verify on a disaggregated follower panics with a false data corruption report:

int __hs_verify_id(WT_SESSION_IMPL *, WT_CURSOR *, WT_CURSOR_BTREE *, uint32_t), 80:
the associated history store key 2 was not found in the data store
file:test_verify_disagg05.wt_stable/WiredTigerCheckpoint.5: WT_PANIC: WiredTiger library panic

__hs_verify_id() sets WT_CONN_DATA_CORRUPTION and raises WT_PANIC, so the process must exit. The data is not corrupt: the leader verifies the same checkpoint cleanly in the same test run. Only the follower panics.

Reproducer

Attached: test_verify_disagg05.py. Deterministic, aborts on every run against unmodified develop.

cd build && python3 ../test/suite/run.py test_verify_disagg05

Root cause: visibility horizon asymmetry between leader and follower

1. Eviction pressure pushes older versions of the keys into the shared history store, and a checkpoint captures them. Note that history store records produced by a checkpoint's own reconciliation only land in the next checkpoint, so a single checkpoint leaves the follower with an empty history store and nothing to check. The reproducer takes several.
2. Every key is deleted at a timestamp. A timestamped tombstone does not clear the key's history store records: __wti_rec_hs_clear_on_tombstone() is only reached when upd_select.no_ts_tombstone is set (src/reconcile/rec_row.c:975).
3. The leader's oldest_timestamp moves past the tombstones and it checkpoints. Reconciliation now drops those keys from the page image entirely. This needs a full image: with leaf_page_delta enabled the key survives as a tombstone in a delta and the problem does not appear.
4. The leader verifies cleanly. The leftover history store records have globally visible stop times from its point of view, so __curhs_next_visible() skips them.
5. The follower reads the same checkpoint with its own, much older, global visibility horizon. In the reproducer the leader's oldest timestamp is 23 and the follower's is 0. The follower does not skip those records, walks history store keys that the data store checkpoint no longer holds, and panics.

The invariant __hs_verify_id() enforces, that every history store key must exist in the data store, only holds for a reader whose visibility horizon matches the one that produced the checkpoint. A follower's does not.

Relationship to WT-18330

This is a distinct defect. It reproduces identically with and without the __wt_hs_verify_one() checkpoint-name change proposed in WT-18330, so it is not caused by that NULL argument. The two share a subsystem and both surface on followers, but they need separate fixes.

Proposed Solution

Directions to consider, in rough order of preference:

• Evaluate obsolescence in __hs_verify_id() against the visibility horizon that produced the checkpoint being verified, rather than the reading connection's current one.
• Alternatively, treat a history store key that is absent from the data store as an error only when the record is not obsolete with respect to the checkpoint's oldest timestamp, and skip it otherwise.
• Failing either, the check cannot be applied safely on a follower and should be skipped there. That loses coverage and should be a last resort.

Whatever the fix, __hs_verify_id() should not set WT_CONN_DATA_CORRUPTION and panic for a condition that the reading connection's own timestamp state can produce. A panic gives the caller no way to distinguish a genuine corruption from this case.

Definition of Done

• test_verify_disagg05.py passes and is committed.
• Leader-side history store verification coverage is unchanged.
• The existing disaggregated verify tests still pass: test_verify_disagg, test_verify_disagg02, test_verify_disagg03.
• If the chosen fix reduces what is checked on a follower, the ticket records exactly what coverage was given up.