-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker, Test Format
-
None
-
Storage Engines - Foundations
-
None
-
5
-
3
A search mismatch failure is still present after WT-16955 fixes. This ticket aims to investigate what could be causing the problem.
Error signature:
table:T00001: WiredTiger trace ID: 16
snapshot-isolation 0000070697.00/op search mismatch
expected {0000070697/LMNOPQRSTUVWXYZABCDE}
found {0000070697/LMNOPQRSTUVWXYZAB}
snapshot-isolation error: dumping to RUNDIR/FAIL.pagedump.1
[role change] follower -> leader
[role change] leader -> follower
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=64896,metadata_checksum=1615e4b5,database_size=235881642,version=1,compatible_version=1],timestamp(hex)=202ed2) ---
[role change] follower -> leader
[role change] leader -> follower
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=98729,metadata_checksum=ab8ae367,database_size=269111893,version=1,compatible_version=1],timestamp(hex)=2d17de) ---
t: FAILED: snap_verify_callback/294: 0
Config:
############################################ # RUN PARAMETERS: V3 ############################################ assert.read_timestamp=0 background_compact=0 background_compact.free_space_target=68 backup=0 backup.incremental=off backup.incr_granularity=15385 backup.live_restore=1 backup.live_restore_read_size=32 backup.live_restore_threads=2 block_cache=0 block_cache.cache_on_checkpoint=1 block_cache.cache_on_writes=0 block_cache.size=27 cache=3072 cache.evict_max=5 cache.eviction_dirty_target=0 cache.eviction_dirty_trigger=0 cache.eviction_updates_target=0 cache.eviction_updates_trigger=0 cache.minimum=0 cache.maximum=0 checkpoint=on checkpoint.log_size=111 checkpoint.wait=85 chunk_cache=0 chunk_cache.capacity=2046 chunk_cache.chunk_size=5 chunk_cache.storage_path=off chunk_cache.type=off compact.free_space_target=73 debug.background_compact=0 debug.checkpoint_retention=10 debug.cursor_reposition=0 debug.eviction=0 debug.log_retention=3 debug.realloc_exact=0 debug.realloc_malloc=0 debug.slow_checkpoint=0 debug.table_logging=0 debug.update_restore_evict=0 disagg.internal_page_delta=1 disagg.leaf_page_delta=1 disagg.multi=0 disagg.multi_validation=0 disagg.enabled=1 disagg.layered=1 disagg.mode=switch disagg.page_log=palite disagg.key_provider=1 disagg.page_log.verbose=0 disagg.drain_threads=7 disk.data_extend=0 disk.encryption=none disk.mmap=1 disk.mmap_all=0 eviction.evict_use_softptr=0 file_manager.close_handle_minimum=20 file_manager.close_idle_time=45 file_manager.close_scan_interval=7 format.abort=0 format.independent_thread_rng=1 format.major_timeout=0 import=0 logging=0 logging.compression=none logging.file_max=170926 logging.prealloc=0 logging.remove=1 obsolete_cleanup.method=off obsolete_cleanup.wait=438 ops.alter=0 ops.compaction=0 ops.hs_cursor=0 ops.pct.modify=0 ops.bound_cursor=0 ops.prepare=0 ops.random_cursor=0 ops.salvage=0 ops.throttle=0 ops.throttle.sleep_us=457066 ops.verify=1 prefetch=1 precise_checkpoint=1 preserve_prepared=1 quiet=1 random.data_seed=13443704 random.extra_seed=15348573 rollback_to_stable_threads=2 runs.in_memory=0 runs.mirror=1 runs.ops=0 runs.predictable_replay=0 runs.source=layered runs.tables=3 runs.threads=32 runs.timer=6 runs.type=row-store runs.verify_failure_dump=0 statistics.mode=fast statistics_log.sources=off stress.aggressive_stash_free=0 stress.aggressive_sweep=0 stress.checkpoint=0 stress.checkpoint_evict_page=0 stress.checkpoint_prepare=0 stress.compact_slow=0 stress.evict_reposition=0 stress.failpoint_eviction_split=1 stress.failpoint_hs_delete_key_from_ts=0 stress.failpoint_rec_before_wrapup=0 stress.hs_checkpoint_delay=0 stress.hs_search=0 stress.hs_sweep=0 stress.prefetch_delay=0 stress.prepare_resolution_1=0 stress.sleep_before_read_overflow_onpage=0 stress.split_1=0 stress.split_2=0 stress.split_3=0 stress.split_4=0 stress.split_5=0 stress.split_6=0 stress.split_7=0 stress.split_8=0 tiered_storage.flush_frequency=0 tiered_storage.storage_source=off transaction.implicit=0 transaction.operation_timeout_ms=2000 transaction.timestamps=1 wiredtiger.config=off wiredtiger.rwlock=1 wiredtiger.leak_memory=0 ############################################ # TABLE PARAMETERS: table 1 ############################################ table1.btree.compression=none table1.btree.dictionary=0 table1.btree.internal_key_truncation=1 table1.btree.internal_page_max=17 table1.btree.key_max=85 table1.btree.key_min=14 table1.btree.leaf_page_max=12 table1.btree.memory_page_max=6 table1.btree.prefix_len=0 table1.btree.prefix_compression=1 table1.btree.prefix_compression_min=6 table1.btree.reverse=0 table1.btree.split_pct=73 table1.btree.value_max=3746 table1.btree.value_min=15 table1.disagg.enabled=0 table1.disk.checksum=on table1.disk.firstfit=0 table1.ops.pareto=0 table1.ops.pareto.skew=41 table1.ops.pct.delete=38 table1.ops.pct.insert=29 table1.ops.pct.read=2 table1.ops.pct.write=31 table1.ops.truncate=1 table1.runs.rows=103724 table1.runs.source=table ############################################ # TABLE PARAMETERS: table 2 ############################################ table2.btree.compression=none table2.btree.dictionary=0 table2.btree.internal_key_truncation=1 table2.btree.internal_page_max=15 table2.btree.key_max=92 table2.btree.key_min=28 table2.btree.leaf_page_max=11 table2.btree.memory_page_max=1 table2.btree.prefix_len=0 table2.btree.prefix_compression=1 table2.btree.prefix_compression_min=7 table2.btree.reverse=0 table2.btree.split_pct=53 table2.btree.value_max=1626 table2.btree.value_min=19 table2.disk.checksum=on table2.disk.firstfit=0 table2.ops.pareto=0 table2.ops.pareto.skew=30 table2.ops.pct.delete=10 table2.ops.pct.insert=27 table2.ops.pct.read=27 table2.ops.pct.write=36 table2.ops.truncate=1 table2.runs.rows=103724 ############################################ # TABLE PARAMETERS: table 3 ############################################ table3.btree.compression=none table3.btree.dictionary=0 table3.btree.internal_key_truncation=1 table3.btree.internal_page_max=14 table3.btree.key_max=66 table3.btree.key_min=19 table3.btree.leaf_page_max=10 table3.btree.memory_page_max=9 table3.btree.prefix_len=0 table3.btree.prefix_compression=1 table3.btree.prefix_compression_min=5 table3.btree.reverse=0 table3.btree.split_pct=72 table3.btree.value_max=3546 table3.btree.value_min=1 table3.disk.checksum=uncompressed table3.disk.firstfit=0 table3.ops.pareto=1 table3.ops.pareto.skew=12 table3.ops.pct.delete=67 table3.ops.pct.insert=30 table3.ops.pct.read=0 table3.ops.pct.write=3 table3.ops.truncate=1 table3.runs.rows=103724 t: process aborting WiredTiger Error: __wt_abort, 29: aborting WiredTiger library