-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Critical - P2
-
None
-
Affects Version/s: None
-
Component/s: Not Applicable
-
Storage Engines - Foundations
-
None
-
None
Branched from the list of 166 BFGs in WT-16531. I have run against 166 BFGs after WT-16531 fix. There is still an data mismatch after step-up as follows:
ubuntu@ip-10-128-246-39:/data/another-wt/wiredtiger/cmake_build/test/format$ ./t -c CONFIG.test
t: process 1101336 running
t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls[role change] leader -> follower
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=991570,metadata_checksum=9d2dcb31,database_size=268013158,version=1,compatible_version=1],timestamp(hex)=36a278) ---
[role change] follower -> leader
mirror: 639392/639449 mismatch: table:T00001: {0000639392.00/opqrstuvwxyzabcdefghijklmnopqrst}/{0000639392/LMNOPQRST}, table:T00002: {0000639449.00/opqrstuvwxy}/{0000639449/LMNO}
last successful match was 639015
mirror error: base cursor (table 1): dumping to RUNDIR/FAIL.pagedump.1
mirror error: table cursor (table 2): dumping to RUNDIR/FAIL.pagedump.2
mirror error: table key number 639449 in table 1: dumping to RUNDIR/FAIL.pagedump.3
mirror error: base key number 639392 in table 2: dumping to RUNDIR/FAIL.pagedump.4
mirror error: base key number 639392 in table 3: dumping to RUNDIR/FAIL.pagedump.5
mirror error: table key number 639449 in table 3: dumping to RUNDIR/FAIL.pagedump.6
t: FAILED: table_verify_mirror/343: base_keyno == table_keyno || (FLD_ISSET(g.trace_flags, TRACE_MIRROR_FAIL) && failures < 20)
The config is as follows:
t: run FAILED ############################################ # RUN PARAMETERS: V3 ############################################ assert.read_timestamp=1 background_compact=0 background_compact.free_space_target=64 backup=0 backup.incremental=off backup.incr_granularity=13122 backup.live_restore=0 backup.live_restore_read_size=4 backup.live_restore_threads=5 block_cache=0 block_cache.cache_on_checkpoint=1 block_cache.cache_on_writes=1 block_cache.size=14 cache=3072 cache.evict_max=2 cache.eviction_dirty_target=0 cache.eviction_dirty_trigger=0 cache.eviction_updates_target=0 cache.eviction_updates_trigger=0 cache.minimum=0 cache.maximum=0 checkpoint=on checkpoint.log_size=186 checkpoint.wait=50 chunk_cache=0 chunk_cache.capacity=4354 chunk_cache.chunk_size=4 chunk_cache.storage_path=off chunk_cache.type=off compact.free_space_target=41 debug.background_compact=0 debug.checkpoint_retention=2 debug.cursor_reposition=1 debug.eviction=0 debug.log_retention=4 debug.realloc_exact=0 debug.realloc_malloc=0 debug.slow_checkpoint=0 debug.table_logging=0 debug.update_restore_evict=0 disagg.internal_page_delta=0 disagg.leaf_page_delta=1 disagg.multi=0 disagg.multi_validation=0 disagg.enabled=1 disagg.layered=1 disagg.mode=switch disagg.page_log=palite disagg.key_provider=1 disagg.page_log.verbose=0 disagg.drain_threads=5 disk.data_extend=0 disk.encryption=none disk.mmap=1 disk.mmap_all=0 eviction.evict_use_softptr=1 file_manager.close_handle_minimum=12 file_manager.close_idle_time=34 file_manager.close_scan_interval=7 format.abort=0 format.independent_thread_rng=1 format.major_timeout=0 import=0 logging=0 logging.compression=none logging.file_max=78025 logging.prealloc=1 logging.remove=1 obsolete_cleanup.method=off obsolete_cleanup.wait=2229 ops.alter=0 ops.compaction=0 ops.hs_cursor=0 ops.pct.modify=0 ops.bound_cursor=0 ops.prepare=0 ops.random_cursor=0 ops.salvage=0 ops.throttle=0 ops.throttle.sleep_us=536607 ops.verify=1 prefetch=1 precise_checkpoint=1 preserve_prepared=1 quiet=1 random.data_seed=8730064 random.extra_seed=16244990 rollback_to_stable_threads=1 runs.in_memory=0 runs.ops=0 runs.predictable_replay=0 runs.source=layered runs.tables=3 runs.threads=1 runs.timer=10 runs.type=row-store runs.verify_failure_dump=0 statistics.mode=all statistics_log.sources=off stress.aggressive_stash_free=0 stress.aggressive_sweep=0 stress.checkpoint=0 stress.checkpoint_evict_page=0 stress.checkpoint_prepare=0 stress.compact_slow=0 stress.evict_reposition=0 stress.failpoint_eviction_split=1 stress.failpoint_hs_delete_key_from_ts=0 stress.failpoint_rec_before_wrapup=0 stress.hs_checkpoint_delay=0 stress.hs_search=0 stress.hs_sweep=0 stress.prefetch_delay=0 stress.prepare_resolution_1=0 stress.sleep_before_read_overflow_onpage=0 stress.split_1=0 stress.split_2=0 stress.split_3=0 stress.split_4=0 stress.split_5=0 stress.split_6=0 stress.split_7=0 stress.split_8=0 tiered_storage.flush_frequency=0 tiered_storage.storage_source=off transaction.implicit=0 transaction.operation_timeout_ms=2000 transaction.timestamps=1 wiredtiger.config=off wiredtiger.rwlock=0 wiredtiger.leak_memory=0 ############################################ # TABLE PARAMETERS: table 1 ############################################ table1.btree.compression=none table1.btree.dictionary=1 table1.btree.internal_key_truncation=1 table1.btree.internal_page_max=14 table1.btree.key_max=117 table1.btree.key_min=32 table1.btree.leaf_page_max=11 table1.btree.memory_page_max=3 table1.btree.prefix_len=0 table1.btree.prefix_compression=0 table1.btree.prefix_compression_min=6 table1.btree.reverse=0 table1.btree.split_pct=97 table1.btree.value_max=3962 table1.btree.value_min=12 table1.disk.checksum=uncompressed table1.disk.firstfit=0 table1.ops.pareto=1 table1.ops.pareto.skew=81 table1.ops.pct.delete=0 table1.ops.pct.insert=98 table1.ops.pct.read=1 table1.ops.pct.write=1 table1.ops.truncate=1 table1.runs.mirror=1 table1.runs.rows=767118 ############################################ # TABLE PARAMETERS: table 2 ############################################ table2.btree.compression=none table2.btree.dictionary=0 table2.btree.internal_key_truncation=1 table2.btree.internal_page_max=14 table2.btree.key_max=31 table2.btree.key_min=25 table2.btree.leaf_page_max=10 table2.btree.memory_page_max=4 table2.btree.prefix_len=0 table2.btree.prefix_compression=1 table2.btree.prefix_compression_min=7 table2.btree.reverse=0 table2.btree.split_pct=94 table2.btree.value_max=3976 table2.btree.value_min=5 table2.disk.checksum=on table2.disk.firstfit=0 table2.ops.pareto=0 table2.ops.pareto.skew=96 table2.ops.pct.delete=61 table2.ops.pct.insert=1 table2.ops.pct.read=25 table2.ops.pct.write=13 table2.ops.truncate=1 table2.runs.mirror=1 table2.runs.rows=767118 ############################################ # TABLE PARAMETERS: table 3 ############################################ table3.btree.compression=none table3.btree.dictionary=0 table3.btree.internal_key_truncation=1 table3.btree.internal_page_max=11 table3.btree.key_max=102 table3.btree.key_min=25 table3.btree.leaf_page_max=12 table3.btree.memory_page_max=10 table3.btree.prefix_len=0 table3.btree.prefix_compression=1 table3.btree.prefix_compression_min=6 table3.btree.reverse=0 table3.btree.split_pct=83 table3.btree.value_max=1580 table3.btree.value_min=5 table3.disk.checksum=on table3.disk.firstfit=0 table3.ops.pareto=0 table3.ops.pareto.skew=79 table3.ops.pct.delete=97 table3.ops.pct.insert=3 table3.ops.pct.read=0 table3.ops.pct.write=0 table3.ops.truncate=1 table3.runs.mirror=1 table3.runs.rows=767118