-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker, Test Format
-
Storage Engines, Storage Engines - Transactions
-
SE Transactions - 2025-09-26
-
5
-
8
format-stress-test-disagg-switch-2 on amazon2023-disagg-stress
Host: i-00eaa122e5221a00b
Project: wiredtiger
Commit: da35aa40
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-switch-2 task_log
Logs:
[1757038507:317943][19604:0xfffefd59eb80], t, file:T00001.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wti_block_disagg_write_internal, 118: WiredTiger assertion failed: 'conn->layered_table_manager.leader'. Trying to write the page from a follower [1757038507:317961][19604:0xfffefd59eb80], t, file:T00001.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-disagg-switch-2 task_log
Logs:
#0 0x0000ffff818bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff818bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff818763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff81862264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff81bc61f8 [PAC] in __wt_abort (session=session@entry=0x116abfc8bcc0) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/os_common/os_abort.c:31 #4 0x0000ffff81a67954 in __wti_block_disagg_write_internal (session=session@entry=0x116abfc8bcc0, block_disagg=0x116abfe2a940, buf=buf@entry=0x116aae3f0f98, block_meta=block_meta@entry=0x116aa79aa6c0, sizep=sizep@entry=0xfffefd59b180, checksump=checksump@entry=0xfffefd59b184, data_checksum=data_checksum@entry=true, checkpoint_io=checkpoint_io@entry=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/block_disagg/block_disagg_write.c:118 #5 0x0000ffff81a67a64 in __wti_block_disagg_write (session=session@entry=0x116abfc8bcc0, block=<optimized out>, buf=buf@entry=0x116aae3f0f98, block_meta=block_meta@entry=0x116aa79aa6c0, addr=addr@entry=0xfffefd59bfb0 "\002", addr_sizep=addr_sizep@entry=0xfffefd59c0b8, data_checksum=data_checksum@entry=true, checkpoint_io=checkpoint_io@entry=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/block_disagg/block_disagg_write.c:243 #6 0x0000ffff81a65514 in __bmd_write (bm=0x116abfe12380, session=0x116abfc8bcc0, buf=0x116aae3f0f98, block_meta=0x116aa79aa6c0, addr=0xfffefd59bfb0 "\002", addr_sizep=0xfffefd59c0b8, data_checksum=true, checkpoint_io=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/block_disagg/block_disagg_mgr.c:109 #7 0x0000ffff81a61088 in __wt_blkcache_write (session=session@entry=0x116abfc8bcc0, buf=buf@entry=0x116aae3f0f98, block_meta=0x116aa79aa6c0, addr=0xfffefd59bfb0 "\002", addr_sizep=0xfffefd59c0b8, compressed_sizep=compressed_sizep@entry=0xfffefd59c0b0, checkpoint=checkpoint@entry=false, checkpoint_io=false, compressed=compressed@entry=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/block_cache/block_io.c:767 #8 0x0000ffff81c00d20 in __rec_write_delta (session=session@entry=0x116abfc8bcc0, r=r@entry=0x116aae3f0c80, chunk=chunk@entry=0x116aae3f0ce8, addr=addr@entry=0xfffefd59bfb0 "\002", addr_sizep=addr_sizep@entry=0xfffefd59c0b8, compressed_sizep=compressed_sizep@entry=0xfffefd59c0b0) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_write.c:2546 #9 0x0000ffff81c07e34 in __rec_split_write (session=session@entry=0x116abfc8bcc0, r=r@entry=0x116aae3f0c80, chunk=0x116aae3f0ce8, last_block=last_block@entry=true) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_write.c:2905 #10 0x0000ffff81c08e54 in __wti_rec_split_finish (session=session@entry=0x116abfc8bcc0, r=r@entry=0x116aae3f0c80) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_write.c:1878 #11 0x0000ffff81bf4300 in __wti_rec_row_leaf (session=session@entry=0x116abfc8bcc0, r=r@entry=0x116aae3f0c80, pageref=pageref@entry=0x116aa278a960, salvage=salvage@entry=0x0) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_row.c:1268 #12 0x0000ffff81c06614 in __reconcile (session=session@entry=0x116abfc8bcc0, ref=ref@entry=0x116aa278a960, salvage=salvage@entry=0x0, flags=flags@entry=5280, page_lockedp=page_lockedp@entry=0xfffefd59cfef) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_write.c:305 #13 0x0000ffff81c0590c in __wt_reconcile (session=session@entry=0x116abfc8bcc0, ref=ref@entry=0x116aa278a960, salvage=salvage@entry=0x0, flags=flags@entry=5280) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/reconcile/rec_write.c:124 #14 0x0000ffff81b927e0 in __evict_reconcile (session=session@entry=0x116abfc8bcc0, ref=ref@entry=0x116aa278a960, evict_flags=evict_flags@entry=0) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/evict/evict_page.c:1044 #15 0x0000ffff81b94814 in __wt_evict (session=session@entry=0x116abfc8bcc0, ref=0x116aa278a960, previous_state=3 '\003', flags=flags@entry=0) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/evict/evict_page.c:236 #16 0x0000ffff81b89ce4 in __evict_page (session=session@entry=0x116abfc8bcc0, is_server=is_server@entry=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/evict/evict_lru.c:2958 #17 0x0000ffff81b8a190 in __evict_lru_pages (session=session@entry=0x116abfc8bcc0, is_server=is_server@entry=false) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/evict/evict_lru.c:1407 #18 0x0000ffff81b8e3ac in __evict_thread_run (session=0x116abfc8bcc0, thread=0x116abfe08820) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/evict/evict_lru.c:354 #19 0x0000ffff81c64764 in __thread_run (arg=0x116abfe08820) at /data/mci/fd34c6466292e9fe1a9a45dea41ac13d/wiredtiger/src/support/thread_group.c:31 #20 0x0000ffff818bdd78 in start_thread () from /lib64/libc.so.6 #21 0x0000ffff8192addc [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts:
