-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Cursors
-
Storage Engines - Foundations
-
472.751
-
None
-
None
-
15
Description
A random cursor on a layered table crashes with SIGFPE when it lands on a leaf page whose disk image was materialized from deltas down to zero entries.
Program terminated with signal SIGFPE, Arithmetic exception. #0 __random_leaf_disk (cbt=0x7f8b1405c2a0, validp=0x7f8ad6be4d2f) at src/btree/bt_random.c:241 241 slot = __wt_random(&cbt->rnd) % entries; #1 __random_leaf (cbt=0x7f8b1405c2a0) at src/btree/bt_random.c:293 #2 __wt_btcur_next_random (cbt=0x7f8b1405c2a0) at src/btree/bt_random.c:627 #3 __wti_curfile_next_random (cursor=0x7f8b1405c2a0) at src/cursor/cur_file.c:221 #4 __clayered_next_random (cursor=0x7f8b142991f0) at src/cursor/cur_layered.c:3230 #5 random_kv (arg=<optimized out>) at test/format/random.c:80
Seen in this patch.
Core dump observations
entries is zero, and the page is internally consistent – not a corrupt or partially-read page.
p cbt->ref->page->entries $1 = 0 p cbt->ref->page->type $2 = 7 '\a' /* WT_PAGE_ROW_LEAF */ p cbt->ref->page->modify $3 = (WT_PAGE_MODIFY *) 0x0 p cbt->ref->page->pg_row $7 = (WT_ROW *) 0x0 p *cbt->ref->page->dsk $5 = { recno = 0, write_gen = 1785, mem_size = 44, u = { entries = 0, datalen = 0 }, type = 7 '\a', flags = 0, reserved = 0, version = 1 } p/x cbt->ref->page->flags_atomic $2 = 0x15 /* BUILD_KEYS | DISK_ALLOC | DISK_SHARED */ p *cbt->ref->page->disagg_info $1 = { old_rec_lsn_max = 4303, rec_lsn_max = 4303, block_meta = { page_id = 864, disagg_lsn = 4303, backlink_lsn = 2265, base_lsn = 2265, checksum = 3171936685, cumulative_size = 3457, delta_count = 1 }, shared_dsk_item = 0x7f8b1c282e70 }
The faulting instruction is divl 0x4(%rsp) – unsigned divide, so SIGFPE means a zero divisor.
pg_row == NULL is the expected companion: with alloc_entries == 0, __wt_page_alloc() deliberately leaves the row array unallocated (bt_page.c:814). A non-zero entries here would itself have been the corruption, since line 242 would then dereference NULL.
delta_count = 1 with base_lsn = 2265 and disagg_lsn = 4303: the image was built by __wti_page_merge_deltas_with_base_image_leaf() from a base image plus one delta, then cached as a shared image (WT_PAGE_DISK_SHARED).
What the page service holds for this page
From the failing run's palite store (RUNDIR.49/kv_home/pages_08.db, table 25, page 864):
lsn backlink_lsn base_lsn flags delta len write_gen entries 2265 1940 0 0 0 3335 1038 6 4303 2265 2265 2 1 122 1785 2
Both persisted components carry entries – no empty image was ever written to the page service. The emptiness is produced by the merge, in cache. This matches reconciliation, which permits a zero-entry chunk only for row-store leaf (rec_write.c:2579) and in disagg jumps to copy_image (rec_write.c:2591), skipping the write block at :2646. Media images are never empty; materialized images may be, so wt verify is unaffected.
The crashing image is the output of this specific merge: bt_page.c:451 sets dsk->write_gen from the last delta, and the core's image has write_gen = 1785 – delta@4303, not base@1038.
Likely mechanism (deduced from the merge code, not directly observed)
The merge loop has exactly three ways an input fails to reach the output:
- a base key whose stop time window is globally visible (bt_page.c:365)
- a delta entry flagged WT_DELTA_LEAF_IS_DELETE, or whose stop is globally visible (bt_page.c:381)
- a base key superseded by a delta entry with the same key (cmp == 0: base is advanced without being packed)
The probable story: this page's keys were deleted, the deletes aged past the oldest timestamp, and materialization stopped emitting them. That is consistent with format doing inserts and removes across a 10k-row keyspace for 50k ops.
Why an empty leaf image is legal
- __wti_page_merge_deltas_with_base_image_leaf() explicitly handles disk_s.entries == 0 when finalizing the header, suppressing the mutually-exclusive EMPTY_V flags (bt_page.c:430). The core's flags = 0 is that guard's output.
- __wt_page_alloc() tolerates alloc_entries == 0 for row-store leaf while asserting non-zero for internal pages (bt_page.c:786 vs :814).
- The delta-materialization read path verifies with WT_VRFY_DISK_EMPTY_PAGE_OK (bt_read.c:382), suppressing the "page has no entries" error.
- Every other row-leaf reader already guards: row_srch.c:615, bt_curprev.c:428, col_srch.c:243, bt_split.c:2054, bt_vrfy.c:1307.
Why this only surfaces now
Before disaggregated storage, dsk != NULL implied bytes off a block device, and those are never empty – the block-level verifiers reject an empty page. Disagg breaks that implication: the image can be synthesized in memory, and a synthesized one may legitimately be empty.
Fix
entries = cbt->ref->page->entries;
+ /* Merging deltas drops keys whose deletes are globally visible, so the image can be empty. */
+ if (entries == 0)
+ return (0);
+
/* This is a relatively cheap test, so try several times. */