-
Type:
Sub-task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Btree
-
None
-
Storage Engines - Persistence
-
206.435
-
StorEng - Defined Pipeline
-
None
Relationship: WT-18636 (the reproducer), WT-15818 (promote the frontier asserts to release checks)
Symptom
A test/format disaggregated run aborts in the eviction path, in __split_multi:
[1792638988:852818][34264:0x7f7e76aba700], t, file:T00002.wt_stable, WT_CURSOR.reset:
[WT_VERB_DEFAULT][ERROR]: __split_multi, 2401: WiredTiger assertion failed:
'page->disagg_info == ((void *)0) || closing || __ref_get_state((ref_new[i])) == 3 ||
(mod->u1.m.multi[i].block_meta != ((void *)0) &&
__wt_materialization_check(session, mod->u1.m.multi[i].block_meta->disagg_lsn))'.
Expression returned false
[1792638988:852854][34264:0x7f7e76aba700], t, file:T00002.wt_stable, WT_CURSOR.reset:
[WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
The assert is src/btree/bt_split.c:2401, in the loop that converts a split page's multiblock reconciliation information into page references:
/*
* A disaggregated child without a retained disk image has been pushed to WT_REF_DISK. Its
* backing block must be behind the materialization frontier so it can be read back safely.
*/
WT_ASSERT(session,
page->disagg_info == NULL || closing || WT_REF_GET_STATE(ref_new[i]) == WT_REF_MEM ||
(mod->mod_multi[i].block_meta != NULL &&
__wt_materialization_check(session, mod->mod_multi[i].block_meta->disagg_lsn)));
Scenario
- A disaggregated (layered) test/format run in disagg.mode=switch, with a materialization frontier published and deliberately lagging behind the workload's last LSN.
- An application thread resets a layered cursor. That releases the page, and the release path decides to evict it: {}clayered_reset_cursors {}
> }}{}{{curfile_reset {}> }}{}{{wt_btcur_reset {}> }}{}{{wt_page_release {}> }}{}{{wt_page_release_evict {}-> }}{_}{}{{{}_wt_evict. - The page is dirty, so eviction takes __evict_page_dirty_update and the reconciled result is a multiblock split.
- __split_multi builds the new references and finds a child whose WT_REF state is WT_REF_DISK (not WT_REF_MEM), with a block_meta->disagg_lsn ahead of the frontier - so the child's backing block is not yet materialized, yet the page has been handed back to disk.
- The assert fires.
Full stack:
#2 __wt_abort src/os_common/os_abort.c:32
#3 __split_multi src/btree/bt_split.c:2401 <-- assert
#4 __split_multi_lock src/btree/bt_split.c:2462
#5 __wt_split_multi src/btree/bt_split.c:2492
#6 __evict_page_dirty_update src/evict/evict_page.c:941
#7 __wt_evict src/evict/evict_page.c:746
#8 __wt_page_release_evict src/btree/bt_read.c:145
#9 __wt_page_release src/include/btree_inline.h:2896
#10 __cursor_reset src/include/cursor_inline.h:310
#11 __wt_btcur_reset src/btree/bt_cursor.c:603
#12 __curfile_reset src/cursor/cur_file.c:281
#13 __clayered_reset_cursors src/cursor/cur_layered.c:2196
#14 __clayered_reset src/cursor/cur_layered.c:2249
#15 table_op test/format/ops.c:1116
#16 ops test/format/ops.c:1513
Why this is a defect
The invariant the assert encodes is the "only readable copy" rule: a disaggregated page whose content exists only in cache must not be handed back to disk until its backing block is materialized, or the next read has nothing to read. A violation is data-loss shaped, not cosmetic.
It is also currently unenforced in release builds: WT_ASSERT compiles to WT_UNUSED when HAVE_DIAGNOSTIC is off (src/include/error.h:347), so a production build would take the same path silently. Promoting the frontier asserts to real checks is WT-15818.
Reproducer
The reproducer is the materialization-frontier support added to test/format by WT-18636 (PR #14994). Without it test/format never publishes a frontier, every frontier-gated decision short-circuits to "allowed", and this path is unreachable.
Command (one iteration of the task; the run that failed was RUNDIR.54 of 54):
./t -c ../../../test/format/CONFIG.disagg -h <rundir> \
disagg.mode=switch runs.rows=10000 runs.ops=50000 cache.maximum=1571 quiet=1
The essential ingredients beyond the stock CONFIG.disagg are:
- disagg.mode=switch - the node steps up and down during the run.
- disagg.frontier_lag=1 / disagg.frontier_lag.max=10000 - publish a frontier that trails the workload's last LSN by up to 10000 LSNs (the WT-18636 knob).
- stress.failpoint_eviction_split=1 - enabled by the randomized config in this run; not known to be required.
- file_manager.close_handle_minimum=0, close_idle_time=1, close_scan_interval=1 and obsolete_cleanup.method=reclaim_space, obsolete_cleanup.wait=1 - also set by WT-18636 when the lag is on.
test/format CONFIG (from the failing RUNDIR.54)
# RUN PARAMETERS: V3 assert.read_timestamp=0 background_compact=0 background_compact.free_space_target=27 backup=0 backup.incremental=off backup.incr_granularity=164 backup.live_restore=0 backup.live_restore_read_size=128 backup.live_restore_threads=9 block_cache=0 block_cache.cache_on_checkpoint=0 block_cache.cache_on_writes=1 block_cache.size=63 btree.reverse=0 cache=3072 cache.evict_max=1 cache.eviction_dirty_target=0 cache.eviction_dirty_trigger=0 cache.eviction_updates_target=0 cache.eviction_updates_trigger=0 cache.minimum=0 cache.maximum=0 checkpoint=on checkpoint.log_size=107 checkpoint.wait=90 checkpoint_threads=3 compact.free_space_target=99 debug.background_compact=0 debug.checkpoint_retention=8 debug.cursor_reposition=0 debug.disagg_slow_truncate_follower=1 debug.eviction=0 debug.log_retention=6 debug.realloc_exact=0 debug.realloc_malloc=0 debug.slow_checkpoint=0 debug.slow_truncate=0 debug.table_logging=0 debug.update_restore_evict=0 disagg.internal_page_delta=1 disagg.leaf_page_delta=1 disagg.multi=0 disagg.multi_validation=0 disagg.enabled=1 disagg.layered=1 disagg.mode=switch disagg.page_log=palite disagg.key_provider=2 disagg.page_log.verbose=0 disagg.drain_threads=11 disagg.preserve=0 disagg.stepdown_async=0 disagg.stepdown_write_mirroring=0 disagg.snapshot_read=0 disagg.victim_cache=1 disagg.victim_cache.size=128 disk.data_extend=0 disk.encryption=rotn-7 disk.mmap=1 disk.mmap_all=0 eviction.evict_use_softptr=0 file_manager.close_handle_minimum=0 file_manager.close_idle_time=1 file_manager.close_scan_interval=1 format.abort=0 format.independent_thread_rng=1 format.major_timeout=0 import=0 logging=0 logging.compression=none logging.file_max=192807 logging.prealloc=0 logging.remove=0 obsolete_cleanup.method=reclaim_space obsolete_cleanup.wait=1 ops.alter=0 ops.compaction=0 ops.hs_cursor=1 ops.bound_cursor=0 ops.prepare=0 ops.reserve=1 ops.random_cursor=0 ops.salvage=0 ops.throttle=0 ops.throttle.sleep_us=657570 ops.verify=1 prefetch=0 prefetch.default=0 precise_checkpoint=1 preserve_prepared=0 quiet=1 random.data_seed=11237233 random.extra_seed=1133696 rollback_to_stable_threads=10 runs.in_memory=0 runs.ops=50000 runs.predictable_replay=0 runs.rows=10000 runs.source=layered runs.tables=3 runs.threads=6 runs.timer=360 runs.type=row-store runs.verify_failure_dump=0 statistics.mode=fast statistics_log.sources=off stress.aggressive_stash_free=0 stress.aggressive_sweep=0 stress.checkpoint=0 stress.checkpoint_evict_page=0 stress.checkpoint_prepare=0 stress.compact_slow=0 stress.disagg_stable_dhandle_delay=0 stress.evict_reposition=0 stress.failpoint_eviction_split=1 stress.failpoint_hs_delete_key_from_ts=0 stress.failpoint_rec_before_wrapup=0 stress.hs_checkpoint_delay=0 stress.hs_search=0 stress.hs_sweep=0 stress.prefetch_delay=0 stress.prepare_resolution_1=0 stress.sleep_before_read_overflow_onpage=0 stress.split_1=0 stress.split_2=0 stress.split_3=0 stress.split_4=0 stress.split_5=0 stress.split_6=0 stress.split_7=0 stress.split_8=0 transaction.implicit=0 transaction.operation_timeout_ms=2000 transaction.timestamps=1 wiredtiger.config=off wiredtiger.rwlock=0 wiredtiger.leak_memory=0 disagg.frontier_lag=1 disagg.frontier_lag.max=10000
The per-table TABLE PARAMETERS blocks (compression, page sizes, per-table op mix, runs.rows=10000) follow in the same file and are unchanged from the artifact.
Evidence
| Item | Value |
|---|---|
| Task | format-stress-test-disagg-switch-pull-request-1 |
| Build variant | ubuntu2004-nonstandalone |
| Project | wiredtiger (mainline; task carries the pull_request tag) |
| Patch | 6ad97bcf2adfcb00076ecd25 (PR #14994) |
| Revision | b4f091741d2f1b513728e2e2919fc604a6ce61ea |
| Task log | Check the comment for details |
| Failing iteration | RUNDIR.54 (54 of 54 runs in that task; 53 succeeded) |
| Core | dump_t.34264.core (uploaded with the task's stacktrace artifact) |
Open questions
- Does it need the lag alone, or lag + role switch? This run used disagg.mode=switch. The next step is an A/B on the same config: disagg.frontier_lag=off versus on, repeated until the abort appears, to show the lag is the trigger.
- Has MongoDB seen this? MongoDB is where the frontier is genuinely published, and
WT-18622in this same area also came from MongoDB rather than WT CI. Worth checking with the disagg owners before calling it novel. - Is the split path expected to handle this? If the design says a split must not push an unmaterialized disagg child to WT_REF_DISK, the fix is in {}split_multi {}/ }}{_}{}{{{}_wt_multi_to_ref. If instead the frontier was published while the node was mid-transition, the fix is in the role-change handling. Determining which needs the A/B above.
- blocks
-
WT-18636 Exercise the materialization frontier in test/format
-
- Blocked
-