SIGFPE in __random_leaf_disk() on a delta-materialized leaf page with zero entries

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cursors

      Description

      A random cursor on a layered table crashes with SIGFPE when it lands on a leaf page whose disk image was materialized from deltas down to zero entries.

      Program terminated with signal SIGFPE, Arithmetic exception.
      #0  __random_leaf_disk (cbt=0x7f8b1405c2a0, validp=0x7f8ad6be4d2f) at src/btree/bt_random.c:241
      241         slot = __wt_random(&cbt->rnd) % entries;
      #1  __random_leaf (cbt=0x7f8b1405c2a0) at src/btree/bt_random.c:293
      #2  __wt_btcur_next_random (cbt=0x7f8b1405c2a0) at src/btree/bt_random.c:627
      #3  __wti_curfile_next_random (cursor=0x7f8b1405c2a0) at src/cursor/cur_file.c:221
      #4  __clayered_next_random (cursor=0x7f8b142991f0) at src/cursor/cur_layered.c:3230
      #5  random_kv (arg=<optimized out>) at test/format/random.c:80
      

      Seen in this patch.

      Core dump observations

      entries is zero, and the page is internally consistent – not a corrupt or partially-read page.

      p cbt->ref->page->entries        $1 = 0
      p cbt->ref->page->type           $2 = 7 '\a'          /* WT_PAGE_ROW_LEAF */
      p cbt->ref->page->modify         $3 = (WT_PAGE_MODIFY *) 0x0
      p cbt->ref->page->pg_row         $7 = (WT_ROW *) 0x0
      p *cbt->ref->page->dsk
        $5 = { recno = 0, write_gen = 1785, mem_size = 44,
               u = { entries = 0, datalen = 0 },
               type = 7 '\a', flags = 0, reserved = 0, version = 1 }
      p/x cbt->ref->page->flags_atomic $2 = 0x15
           /* BUILD_KEYS | DISK_ALLOC | DISK_SHARED */
      p *cbt->ref->page->disagg_info
        $1 = { old_rec_lsn_max = 4303, rec_lsn_max = 4303,
               block_meta = { page_id = 864, disagg_lsn = 4303,
                              backlink_lsn = 2265, base_lsn = 2265,
                              checksum = 3171936685, cumulative_size = 3457,
                              delta_count = 1 },
               shared_dsk_item = 0x7f8b1c282e70 }
      

      The faulting instruction is divl 0x4(%rsp) – unsigned divide, so SIGFPE means a zero divisor.

      pg_row == NULL is the expected companion: with alloc_entries == 0, __wt_page_alloc() deliberately leaves the row array unallocated (bt_page.c:814). A non-zero entries here would itself have been the corruption, since line 242 would then dereference NULL.

      delta_count = 1 with base_lsn = 2265 and disagg_lsn = 4303: the image was built by __wti_page_merge_deltas_with_base_image_leaf() from a base image plus one delta, then cached as a shared image (WT_PAGE_DISK_SHARED).

      What the page service holds for this page

      From the failing run's palite store (RUNDIR.49/kv_home/pages_08.db, table 25, page 864):

      lsn   backlink_lsn  base_lsn  flags  delta  len   write_gen  entries
      2265  1940          0         0      0      3335  1038       6
      4303  2265          2265      2      1      122   1785       2
      

      Both persisted components carry entries – no empty image was ever written to the page service. The emptiness is produced by the merge, in cache. This matches reconciliation, which permits a zero-entry chunk only for row-store leaf (rec_write.c:2579) and in disagg jumps to copy_image (rec_write.c:2591), skipping the write block at :2646. Media images are never empty; materialized images may be, so wt verify is unaffected.

      The crashing image is the output of this specific merge: bt_page.c:451 sets dsk->write_gen from the last delta, and the core's image has write_gen = 1785 – delta@4303, not base@1038.

      Likely mechanism (deduced from the merge code, not directly observed)

      The merge loop has exactly three ways an input fails to reach the output:

      1. a base key whose stop time window is globally visible (bt_page.c:365)
      2. a delta entry flagged WT_DELTA_LEAF_IS_DELETE, or whose stop is globally visible (bt_page.c:381)
      3. a base key superseded by a delta entry with the same key (cmp == 0: base is advanced without being packed)

      The probable story: this page's keys were deleted, the deletes aged past the oldest timestamp, and materialization stopped emitting them. That is consistent with format doing inserts and removes across a 10k-row keyspace for 50k ops.

      Why an empty leaf image is legal

      • __wti_page_merge_deltas_with_base_image_leaf() explicitly handles disk_s.entries == 0 when finalizing the header, suppressing the mutually-exclusive EMPTY_V flags (bt_page.c:430). The core's flags = 0 is that guard's output.
      • __wt_page_alloc() tolerates alloc_entries == 0 for row-store leaf while asserting non-zero for internal pages (bt_page.c:786 vs :814).
      • The delta-materialization read path verifies with WT_VRFY_DISK_EMPTY_PAGE_OK (bt_read.c:382), suppressing the "page has no entries" error.
      • Every other row-leaf reader already guards: row_srch.c:615, bt_curprev.c:428, col_srch.c:243, bt_split.c:2054, bt_vrfy.c:1307.

      Why this only surfaces now

      Before disaggregated storage, dsk != NULL implied bytes off a block device, and those are never empty – the block-level verifiers reject an empty page. Disagg breaks that implication: the image can be synthesized in memory, and a synthesized one may legitimately be empty.

      Fix

           entries = cbt->ref->page->entries;
      
      +    /* Merging deltas drops keys whose deletes are globally visible, so the image can be empty. */
      +    if (entries == 0)
      +        return (0);
      +
           /* This is a relatively cheap test, so try several times. */
      

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Wei Hu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: