Summary

session->pf.prefetch_disk_read_count is a single per-session counter used by __wt_session_prefetch_check to infer that a session is scanning a btree it does not have in cache. A session that interleaves reads across two dhandles - one cold, one cache-resident - can never build the streak the heuristic requires, so pre-fetch never triggers even though the cold dhandle is being scanned end to end.

This limitation is already documented in the code, immediately above the counter in src/btree/bt_read.c:

It might not work if a session has multiple cursors on different tables open, since the operations on different tables get in the way of the heuristic. That isn't super likely - this is to catch traversals through a btree, not complex multi-table user transactions.

The disaggregated checkpoint pick-up is exactly that case, and it is not unlikely - it happens on every follower pick-up.

Why the counter cannot climb

The rule in bt_read.c, applied for every leaf page the session brings into scope:

if (F_ISSET_ATOMIC_16(page, WT_PAGE_PREFETCH) || read_from_disk)
    ++session->pf.prefetch_disk_read_count;
else
    session->pf.prefetch_disk_read_count = 0;

A cache hit does not decrement the counter, it zeroes it. The gate needs the counter to reach 2, so the heuristic is really counting consecutive disk reads with no intervening cache hit.

_disagg_apply_checkpoint_meta drives eight cursors on one session: four over the shared metadata (remote, every leaf is a page log round trip) and four over the local metadata (small, hot, every leaf is a cache hit). On top of that, each table processed performs a md_write_cursor->search() and several _disagg_insert_meta calls against the local metadata.

Measured on disagg_measure_startup_primary_25k_locust with 50,029 tables: roughly 6,700 shared-metadata leaf reads, i.e. about 7.5 tables per shared-metadata page. So each single increment is followed by dozens of local-metadata cache hits before the next shared page is read. The counter oscillates 1 -> 0 -> 1 and is never 2 when pre-fetch asks.

Raising the threshold would not help - no value above 1 is reachable, because the streak does not survive a single table.

Evidence

Instrumented runs on disagg_measure_startup_primary_25k_locust, startup pick-up, 50,029 tables. Variant B bypasses only the internal-session check, leaving the disk-read heuristic active:

In variant B the heuristic refuses 94.0% of checks (6,315 of 6,716), and prefetch_disk_one is 6,312 - the counter was sitting at exactly 1 almost every time it was consulted.

The 304 checks that did survive queued 2,969 pages (about 9.8 each, since a trigger queues up to WT_PREFETCH_QUEUE_PER_TRIGGER), so per-trigger depth partially masks the problem. That is why variant B still recovers some of the win rather than none.

Note the A to B apply-time gap (1,378 ms) is within the ~7% run-to-run drift observed on this workload, so treat the counter evidence as the reliable signal and the millisecond split as indicative.

Proposed approach

Scope the counter to the dhandle whose pages are being read, rather than to the session. The cold dhandle then accumulates an unbroken run of disk reads and trips the threshold on its second page, while the cache-resident dhandle stays at 0 - which is the correct answer for it.

Implementation is open. session->pf holds a single uint64_t today; a small fixed-size set of (dhandle, count) slots in WT_PREFETCH would cover the realistic number of concurrently scanned dhandles without unbounded growth. Alternatives worth weighing: keying off session->dhandle at the point of update, or moving the counter to the btree cursor.

Tasks

• Reproduce the pinned counter in a test that scans a cold table while touching a cache-resident one on the same session, asserting prefetch_skipped_disk_read_count is non-zero before the change.
• Scope prefetch_disk_read_count per dhandle, bounding any per-session storage added.
• Confirm the heuristic now triggers for the cold dhandle and still does not trigger for the cache-resident one.
• Re-run disagg_measure_startup_primary_25k_locust with only the internal-session check bypassed and confirm the disk-read refusals drop from ~94% to near zero.
• Validate no pre-fetch regression on workloads the current heuristic already serves - this changes pre-fetch behaviour globally, not only for disaggregated storage.

Definition of done

• The disk-read heuristic triggers for a session scanning a cold dhandle while interleaving cache hits on another dhandle.
• A test covers the interleaved case and fails without the change.
• Instrumented run shows prefetch_skipped_disk_read_count near zero for the checkpoint pick-up scan with no disk-read bypass in place.
• No regression on existing pre-fetch performance coverage.
• The stale caveat in the bt_read.c comment is updated or removed.

Out of scope

The internal-session check in __wt_session_prefetch_check is a separate and categorical refusal - it rejected 7,129 of 7,129 checks in variant A. Callers that legitimately scan on an internal session still need their own opt-in; that is handled in WT-18472 and is not addressed here.