-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines, Storage Engines - Transactions
-
539.956
-
None
-
None
-
5
format-stress-test-disagg-switch-data-validation-2 on amazon2023-arm64-release-nonstandalone
Host: i-0d3a293d34c9d9998
Project: wiredtiger
Commit: c5d050f4
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
[377/672] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o [579/672] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
[1781181792:010383][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __layered_drain_worker_run, 837: Failed to clear ingest table "file:T00003.wt_ingest": WT_ROLLBACK: conflict between concurrent operations
[1781181792:010452][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010456][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010459][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010462][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010464][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010466][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010468][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010470][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010472][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010474][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010475][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010478][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010479][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010481][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010485][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010487][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010489][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010491][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010493][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010495][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010497][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010499][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010501][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010503][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010505][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010507][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010509][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010511][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010513][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010516][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010518][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010520][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010522][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1781181792:010524][13406:0xffffb3aedbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
t: FAILED: disagg_switch_roles/255: g.wts_conn->reconfigure(g.wts_conn, "disaggregated=(role=leader)"): WT_PANIC: WiredTiger library panic
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
[1781181792:857611][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857644][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857647][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857649][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857651][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857653][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857655][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857657][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857659][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857665][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857667][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857669][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1781181792:857671][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 197: Error at src/cursor/cur_stat.c:197: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
[1781181792:857673][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __statlog_dump, 379: Error at src/conn/conn_stat.c:379: "ret" failed: WT_PANIC: WiredTiger library panic
[1781181792:857676][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_close, 342: Error at src/cursor/cur_stat.c:342: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
[1781181792:857678][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __statlog_dump, 386: Error at src/conn/conn_stat.c:386: "cursor->close(cursor)" failed: WT_PANIC: WiredTiger library panic
[1781181792:857680][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __statlog_log_one, 452: Error at src/conn/conn_stat.c:452: "__statlog_dump(session, conn->home, true)" failed: WT_PANIC: WiredTiger library panic
[1781181792:857682][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __statlog_server, 546: Error at src/conn/conn_stat.c:546: "__statlog_log_one(session, &path, &tmp)" failed: WT_PANIC: WiredTiger library panic
[1781181792:857684][13406:0xffffb9badbc0], t, statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 551: statistics log server error: WT_PANIC: WiredTiger library panic
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
t: process aborting
WiredTiger Error: __wt_abort, 29: aborting WiredTiger library
Error at src/conn/conn_reconfig.c:449: "__wti_disagg_conn_config(session, cfg, true)" failed with WT_PANIC: WiredTiger library panic (-31804)
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
test/format run configuration highlights
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
#0 0x0000ffffbccc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffbccc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffbcc7b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffbcc62224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffffbce4f574 [PAC] in __wt_abort (session=session@entry=0x0) at /data/mci/da1415f0adff9518be8938a683ac73fa/wiredtiger/src/os_common/os_abort.c:32 #4 0x000000000042367c in testutil_die (e=-31804, fmt=fmt@entry=0x425838 "%s/%d: %s") at /data/mci/da1415f0adff9518be8938a683ac73fa/wiredtiger/test/utility/misc.c:75 #5 0x000000000040cc90 in disagg_switch_roles () at /data/mci/da1415f0adff9518be8938a683ac73fa/wiredtiger/test/format/format_disagg.c:255 #6 0x00000000004031e0 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/da1415f0adff9518be8938a683ac73fa/wiredtiger/test/format/t.c:410
Repro Artifacts: