With the PALite victim cache enabled, test/format disagg switch-mode runs fail stable table verify with the child page's write generation newer than its parent's:

[...] file:T00001.wt_stable, WT_SESSION.verify: __verify_tree, 1070: child write generation number
652658 is greater/equal to the parent page write generation number 562621: Invalid argument
[...] layered:T00001, WT_SESSION.verify: __schema_layered_stable_worker_verify, 143: Verify
(layered): file:T00001.wt_stable stable table verification failed on the leader: Invalid argument
t: FAILED: table_verify/71

The invariant is that a parent reconciles only after its children, so the parent's write generation is always higher. Both sides of the comparison are disk images, so verify is assembling a tree from two different points in time.

Patch Results/Failure Occurrences 

• victim cache pinned on - 1 failure / 48 task-runs
• victim cache pinned off - 0 failures / 48 task-runs
• unmodified develop - 0 failures / 48 task-runs

Plus 3 earlier failures with the same signature on randomized runs where the knob rolled on. So 4 occurrences, all with the cache enabled, none across 96 task-runs without it.

Scope

The victim cache only ever serves reads - cache_put() writes to an in-memory map, and the real put() erases the cache entry before writing to storage - so the page log contents are not directly damaged, and the same runs pass with the cache off. But a wrong page image served to the read path could be modified and reconciled back out, so durable damage is not excluded. Verify catches it here; an ordinary read has no equivalent check.

Hypothesis, unconfirmed

The cache is keyed {page_id, lsn} and is per page log handle. If a handle survives a role switch with entries still in it, a {page_id, lsn from the previous role could resolve to content that no longer belongs with the tree being read. Worth checking whether handles are closed and reopened on a switch, and whether lsns are unique across one.

Reproduction

Long disagg switch runs, ~2h, verify catches it at the end. Four independent seeds:

data_seed=15246013  extra_seed=5447301
data_seed=10368247  extra_seed=12013546
data_seed=10129241  extra_seed=11032113
data_seed=15328624  extra_seed=6928271

all disagg.mode=switch, runs.source=layered, row-store, 3 tables, ops.verify=1, cache >= 3072MB, with disagg.victim_cache=1. Entry counts at failure ranged 85 to 9709, so it is not specific to a small or large cache.