Issue Summary

_verify_key_hs() opens and closes a history store cursor for every key it checks. It is called from the per-cell loop in _verify_page_cells() (src/btree/bt_vrfy.c), so verifying an N-key tree performs N history store cursor opens and closes. Each open callocs a WT_CURSOR_HS, allocates a scratch buffer and opens a file cursor on the history store; each close frees all of it.

Measurement

20,000 keys, two committed versions each, non-disaggregated, diagnostic build:

per-key HS check ON          wall=0.132s  open_time_internal=34871us
per-key HS check OFF         wall=0.021s  open_time_internal=2us

ON is the default; OFF is verify(uri, "skip_per_key_hs=true").

That is roughly 1.7 microseconds per key in cursor open and close alone: about a third of the per-key check's cost, and about a quarter of total verify wall time in this run. It scales linearly with key count, so on a large collection it is minutes of pure cursor churn.

cursor_open_count reads zero throughout, because it only counts application cursor opens. Internal opens are timed but not counted, which is part of why this has been easy to miss.

Why it matters: this may reduce the WT-10779 regression

WT-10779 resurrected the extended per-key history store validation. [BF-44595] then identified a performance regression from it, WT-18094 added the skip_per_key_hs flag to bypass the check, and SERVER-131938 opted the server out entirely. SERVER-131939 tracks eventually turning it back on, possibly only for a sampled fraction of runs.

Removing the per-key cursor open cuts roughly a third of the check's overhead without changing what it validates, so it may make the check cheap enough to re-enable more widely and could unblock SERVER-131939.

It is unlikely to be a complete answer on its own. The remaining cost is the per-key history store search and backwards iteration, which is inherent to what the check does. Treat the third as the cheapest available win, not as the whole gap.

Proposed Solution

Hoist the cursor out of the per-key path: open it once per tree, keep it on WT_VSTUFF, and reposition it per key using the __wt_curhs_search_near_before() the code already performs. Close it when the checkpoint's verification finishes.

Reusing a history store cursor across keys is well precedented. _wti_rec_hs_clear_on_tombstone() keeps r->hs_cursor across keys and re-points it at a different btree ID with _wt_curhs_set_btree_id() rather than reopening (src/reconcile/rec_write.c).

WT-18330 introduced __wt_hs_verify_cursor_open(), now the single place both verify call sites obtain their cursor, so the lifetime change has one natural home.

Definition of Done

• Verify performs a bounded number of history store cursor opens per tree rather than one per key.
• The per-key validation still checks the same keys and still catches the same problems: test_verify3, test_verify_hs_overlap, test_verify_disagg through test_verify_disagg04 pass.
• A before and after measurement of open_time_internal and verify wall time on a large tree is recorded here.
• SERVER-131939 is updated with the resulting cost, so the server side can decide whether the check can be re-enabled and at what sampling rate.