ExportXMLWordPrintableJSON

      format-stress-test-disagg-switch-data-validation-stepdown-async-1 on ubuntu2004-stress-tests-arm64, mainline at e5e7a1366e (linked in comment), the first run of this task after WT-18657 turned the victim cache on for it. Aborted on:

      __random_leaf_disk, 245: WiredTiger assertion failed:
      'page->disagg_info != ((void *)0) && page->disagg_info->block_meta.delta_count > 0'
      

      That check is WT-18198's fix: a leaf page that was read in with zero entries must have come from merging deltas into a base image. A stored page image always has at least one entry.

      How the victim cache caused it

      a page whose keys are all deleted
              |
      reconciliation: nothing survives onto the page
              |
      an EMPTY image is kept in memory only        <- legitimate (skip-write:
         never written to the page log, rec_write.c:2769)
              |
      eviction: the victim cache stores that
         empty image under the stored page's identity
         (the page_id + lsn address of the page's real image in the page log)
              |
      a later read: cache hit - the cache serves the
         empty image as if it were the stored page's
         (the page's real, non-empty image in the page log)
              |
      the reader materializes a 0-entry page,
      the assertion rejects it
      

      Evidence, from the task artifacts

      • The page log never stores empty images: scanned every shard (in <ARTEFACTS_FOLDER>/cmake_build/test/format/RUNDIR.3/kv_home/pages_*.db) - 322,986 non-delta stored images, zero with zero entries. The served image did not come from the page log.
      • The run had deltas off (disagg.leaf_page_delta=0), so a read can't drop entries by merging - a plain page-log read cannot produce an empty page either.
      • The failing page was read in and never modified - the assertion fired on the "page as we read it from disk" branch in __random_leaf.
      • The cache was hot in that iteration: 820 pages added, 257 by application threads.
      • WT-18804's fix confirms the design: "the cache stores the image under the backing block's identity".

      So this isn't the cache exposing a broken invariant - the invariant still holds for everything the page log stores. The cache manufactured a page state that can't otherwise exist, by serving a never-written image through the slot that means "this is what the page log stores".

      Proposed fix

      Refuse to cache an empty image that would be served as a standalone base image, at the eligibility gate. A standalone image is never stored empty - only a merged base-and-delta image can legitimately be empty, and its delta count says so:

      if (disk_image->write_gen != __wt_page_write_gen(page))
          return (WTI_EVICT_VICTIM_NO_IMAGE);
      
      /*
       * Only a merged base-and-delta image can legitimately be empty, and its delta count says so. A
       * standalone image is never stored empty, so an empty one must not stand in for the block.
       */
      if (disk_image->u.entries == 0 && page->disagg_info->block_meta.delta_count == 0)
          return (WTI_EVICT_VICTIM_EMPTY_IMAGE);
      

      The check mirrors the read-side assertion: it refuses to cache exactly the page shape the assertion rejects, reading the same field. Merged-empty pages (deltas on) stay cacheable.

      Repro

      Probabilistic. Seeds random.data_seed=16525874, random.extra_seed=11505065; the failing iteration's CONFIG is in the task artifacts (table 3 at 94% deletes with stress.failpoint_hs_delete_key_from_ts=1, disagg.victim_cache=1, deltas off). Expect it to take a few runs.

            Assignee:
            Mariam Mojid
            Reporter:
            Mariam Mojid
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: