-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Block Cache
-
Storage Engines - Persistence
-
203.925
-
StorEng - Defined Pipeline
-
None
-
15
format-stress-test-disagg-switch-data-validation-stepdown-async-1 on ubuntu2004-stress-tests-arm64, mainline at e5e7a1366e (linked in comment), the first run of this task after WT-18657 turned the victim cache on for it. Aborted on:
__random_leaf_disk, 245: WiredTiger assertion failed: 'page->disagg_info != ((void *)0) && page->disagg_info->block_meta.delta_count > 0'
That check is WT-18198's fix: a leaf page that was read in with zero entries must have come from merging deltas into a base image. A stored page image always has at least one entry.
How the victim cache caused it
a page whose keys are all deleted
|
reconciliation: nothing survives onto the page
|
an EMPTY image is kept in memory only <- legitimate (skip-write:
never written to the page log, rec_write.c:2769)
|
eviction: the victim cache stores that
empty image under the stored page's identity
(the page_id + lsn address of the page's real image in the page log)
|
a later read: cache hit - the cache serves the
empty image as if it were the stored page's
(the page's real, non-empty image in the page log)
|
the reader materializes a 0-entry page,
the assertion rejects it
Evidence, from the task artifacts
- The page log never stores empty images: scanned every shard (in <ARTEFACTS_FOLDER>/cmake_build/test/format/RUNDIR.3/kv_home/pages_*.db) - 322,986 non-delta stored images, zero with zero entries. The served image did not come from the page log.
- The run had deltas off (disagg.leaf_page_delta=0), so a read can't drop entries by merging - a plain page-log read cannot produce an empty page either.
- The failing page was read in and never modified - the assertion fired on the "page as we read it from disk" branch in __random_leaf.
- The cache was hot in that iteration: 820 pages added, 257 by application threads.
WT-18804's fix confirms the design: "the cache stores the image under the backing block's identity".
So this isn't the cache exposing a broken invariant - the invariant still holds for everything the page log stores. The cache manufactured a page state that can't otherwise exist, by serving a never-written image through the slot that means "this is what the page log stores".
Proposed fix
Refuse to cache an empty image that would be served as a standalone base image, at the eligibility gate. A standalone image is never stored empty - only a merged base-and-delta image can legitimately be empty, and its delta count says so:
if (disk_image->write_gen != __wt_page_write_gen(page)) return (WTI_EVICT_VICTIM_NO_IMAGE); /* * Only a merged base-and-delta image can legitimately be empty, and its delta count says so. A * standalone image is never stored empty, so an empty one must not stand in for the block. */ if (disk_image->u.entries == 0 && page->disagg_info->block_meta.delta_count == 0) return (WTI_EVICT_VICTIM_EMPTY_IMAGE);
The check mirrors the read-side assertion: it refuses to cache exactly the page shape the assertion rejects, reading the same field. Merged-empty pages (deltas on) stay cacheable.
Repro
Probabilistic. Seeds random.data_seed=16525874, random.extra_seed=11505065; the failing iteration's CONFIG is in the task artifacts (table 3 at 94% deletes with stress.failpoint_hs_delete_key_from_ts=1, disagg.victim_cache=1, deltas off). Expect it to take a few runs.
- is related to
-
WT-18198 SIGFPE in __random_leaf_disk() on a delta-materialized leaf page with zero entries
-
- Closed
-
-
WT-18804 [disagg] victim cache: child write generation inversion in stable table verify under role switch
-
- Closed
-
-
WT-18657 Add PALite block cache to test/format
-
- Closed
-
-
WT-18925 test_layered_victim_cache02 step-down: eviction did not put a page in the victim cache (unit-test, amazon2023-arm64)
-
- Open
-