Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-test-disagg-switch-1
First Failing Revision:
- d8cd17c2bf745d337a0bd57561604bcd643cf934(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15556
Count of Linked BFGs (Last 30 days):
1

format-stress-test-disagg-switch-1 on amazon2023-disagg-stress

Host: i-04db4bd20e693a7b3
Project: wiredtiger
Commit: d8cd17c2
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-1

format-stress-test-disagg-switch-1 task_log

Logs:

[1758422512:372159][4267:0xffff99674040], t, layered:T00001, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00001.wt_stable stable table verification failed. : No such file or directory
[1758422512:372359][4267:0xffff99674040], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00002.wt_stable stable table verification failed. : No such file or directory
[1758422512:487743][4267:0xffff99674040], t, layered:T00003, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00003.wt_stable stable table verification failed. : No such file or directory
[1758422523:646275][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __layered_copy_ingest_table, 1632: WiredTiger assertion failed: 'prev_upd->type == 3'. Expression returned false
[1758422523:646428][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1758422523:646433][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646436][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646438][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646439][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646441][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646442][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646444][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646445][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646446][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
[1758422523:646448][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1758422523:646450][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1758422523:646451][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1758422523:646453][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1758422523:646454][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646456][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646459][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646460][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646462][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646463][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646464][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646466][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646467][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646468][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646470][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646471][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1758422523:646472][4267:0xffff99674040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found

logs

format-stress-test-disagg-switch-1 task_log

Logs:

#0  0x0000ffff97cbf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff97cbf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff97c763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff97c62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff9801c714 [PAC] in __wt_abort (session=session@entry=0x31333fc7fe80) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff97f492e0 in __layered_copy_ingest_table (session=0x31333fc7fe80, entry=entry@entry=0x31333fe2b440) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/conn/conn_layered.c:1632
#5  0x0000ffff97f4a3a0 in __layered_drain_ingest_tables (session=session@entry=0x31333fc7a000) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/conn/conn_layered.c:1750
#6  0x0000ffff97f4f320 in __wti_disagg_conn_config (session=session@entry=0x31333fc7a000, cfg=cfg@entry=0xffffecf0fb08, reconfig=reconfig@entry=true) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/conn/conn_layered.c:1122
#7  0x0000ffff97f57ad8 in __wti_conn_reconfig (session=session@entry=0x31333fc7a000, cfg=cfg@entry=0xffffecf0fb08) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/conn/conn_reconfig.c:450
#8  0x0000ffff97f35f5c in __conn_reconfigure (wt_conn=<optimized out>, config=<optimized out>) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/src/conn/conn_api.c:1354
#9  0x000000000040b9e8 in disagg_switch_roles () at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/test/format/format_disagg.c:54
#10 0x000000000041a8cc in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/4abee89578e9f2c88e10b933a6b36136/wiredtiger/test/format/t.c:398

logs

Repro Artifacts:

CONFIG

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Sep 22 2025 07:10:37 AM UTC
Updated:: Sep 22 2025 11:17:24 PM UTC
Resolved:: Sep 22 2025 11:17:24 PM UTC

Details

Description

Attachments

Activity

People

Dates