-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
format-stress-test-disagg-switch-data-validation-1 on amazon2023-stress-tests-arm64
Host: i-00a1efbac521397ad
Project: wiredtiger
Commit: e7dfa0d1
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-switch-data-validation-1 task_log
Logs:
[1778839430:781842][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:1842: Pages::verify_chain(const std::vector<PageInfo>&, uint64_t)::<lambda(const PageInfo&)>: Full page backlink_lsn mismatch: {table_id=33, page_id=260069, lsn=2473046, backlink_lsn=2398100, base_lsn=0, flags=0x0}, expected: 0
[1778839430:781923][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:575: int safe_call(WT_SESSION*, S*, MemberFunc, Args&& ...) [with T = PaliteHandle; S = __wt_page_log_handle; MemberFunc = int (PaliteHandle::*)(long unsigned int, long unsigned int, __wt_page_log_put_args*, const __wt_item*); Args = {long unsigned int&, long unsigned int&, __wt_page_log_put_args*&, const __wt_item*&}; WT_SESSION = __wt_session]: Call failed
[1778839430:781936][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __checkpoint_tree, 2941: checkpoint failed during cache flush and reconciliation: Invalid argument
[1778839430:781949][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2574: Error at src/reconcile/rec_write.c:2574: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: Invalid argument
[1778839430:781953][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_sync_file, 414: Error at src/btree/bt_sync.c:414: "__wt_reconcile(session, walk, NULL, rec_flags)" failed: Invalid argument
[1778839430:781956][27715:0xffff87601040], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_tree, 2941: Error at src/checkpoint/checkpoint_txn.c:2941: "ret" failed: Invalid argument
[1778839430:781994][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write_internal, 186: Error at src/block_disagg/block_disagg_write.c:186: "plhandle->plh_put(plhandle, &session->iface, page_id, 0, &put_args, buf)" failed: Invalid argument
[1778839430:782000][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write, 246: Error at src/block_disagg/block_disagg_write.c:246: "__wti_block_disagg_write_internal(session, block_disagg, buf, block_meta, page_image_size, &size, &checksum, data_checksum, checkpoint_io)" failed: Invalid argument
[1778839430:782003][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_blkcache_write, 843: Error at src/block_cache/block_io.c:843: "checkpoint ? bm->checkpoint(bm, session, ip, block_meta, btree->ckpt, data_checksum) : bm->write(bm, session, ip, block_meta, page_image_size, addr, addr_sizep, data_checksum, checkpoint_io)" failed: Invalid argument
[1778839430:782006][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write, 1054: Error at src/reconcile/rec_write.c:1054: "__wt_blkcache_write(session, buf, block_meta, buf->size, addr, addr_sizep, compressed_sizep, checkpoint, checkpoint_io, compressed)" failed: Invalid argument
[1778839430:782008][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write_image, 2292: Error at src/reconcile/rec_write.c:2292: "__rec_write(session, &chunk->image, multi->block_meta, addr, addr_sizep, compressed_sizep, false, F_ISSET(r, WT_REC_CHECKPOINT), false)" failed: Invalid argument
[1778839430:782010][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2574: Error at src/reconcile/rec_write.c:2574: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: Invalid argument
[1778839430:782012][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_sync_file, 414: Error at src/btree/bt_sync.c:414: "__wt_reconcile(session, walk, NULL, rec_flags)" failed: Invalid argument
[1778839430:782014][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_tree, 2941: Error at src/checkpoint/checkpoint_txn.c:2941: "ret" failed: Invalid argument
[1778839430:782016][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_apply_to_dhandles, 367: Error at src/checkpoint/checkpoint_txn.c:367: "ret" failed: Invalid argument
[1778839430:782018][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_selected_dhandles, 3212: Error at src/checkpoint/checkpoint_txn.c:3212: "__checkpoint_apply_to_dhandles(session, cfg, __checkpoint_tree_helper)" failed: Invalid argument
[1778839430:782020][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_db_internal, 1711: Error at src/checkpoint/checkpoint_txn.c:1711: "__checkpoint_selected_dhandles(session, cfg)" failed: Invalid argument
[1778839430:782023][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __checkpoint_db_internal, 1909: Disaggregated storage checkpoint failed, unable to rollback, panic to avoid corruption: Invalid argument
[1778839430:782025][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __checkpoint_db_internal, 1909: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1778839430:782028][27715:0xffff87601040], t, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-disagg-switch-data-validation-1 task_log
Logs:
#0 0x0000ffff85cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff85cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff85c7b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff85c62224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff8604a234 [PAC] in __wt_abort (session=session@entry=0x7151aad3bf00) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffff860f4490 in __wt_panic_func (session=session@entry=0x7151aad3bf00, error=error@entry=22, func=func@entry=0xffff861e6c10 <__PRETTY_FUNCTION__.38> "__checkpoint_db_internal", line=line@entry=1909, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff861607d8 "Disaggregated storage checkpoint failed, unable to rollback, panic to avoid corruption") at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/support/err.c:633 #5 0x0000ffff85f37104 in __checkpoint_db_internal (session=session@entry=0x7151aad3bf00, cfg=cfg@entry=0xffffcd57ea38) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/checkpoint/checkpoint_txn.c:1909 #6 0x0000ffff85f37558 in __checkpoint_db_wrapper (session=session@entry=0x7151aad3bf00, cfg=cfg@entry=0xffffcd57ea38) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/checkpoint/checkpoint_txn.c:1968 #7 0x0000ffff85f37934 in __wt_checkpoint_db (session=session@entry=0x7151aad3bf00, cfg=cfg@entry=0xffffcd57ea38, waiting=waiting@entry=true) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/checkpoint/checkpoint_txn.c:2047 #8 0x0000ffff860d188c in __session_checkpoint (wt_session=0x7151aad3bf00, config=<optimized out>) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/src/session/session_api.c:2439 #9 0x000000000040d068 in disagg_switch_roles () at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/test/format/format_disagg.c:260 #10 0x000000000041bbd8 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/a875a9d66fd187910249c1761df2b7a6/wiredtiger/test/format/t.c:410
Repro Artifacts: