Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Transactions - 2025-11-07
Story Points:
5
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-test-disagg-switch-1
First Failing Revision:
- caf29b5fb78ca0f81a52a4b04dadce992704c48c(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15699
Count of Linked BFGs (Last 30 days):
1

format-stress-test-disagg-switch-1 on amazon2023-disagg-stress

Host: i-00a0159216385e7f1
Project: wiredtiger
Commit: caf29b5f
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-1

format-stress-test-disagg-switch-1 task_log

Logs:

[1759794889:910390][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __rec_hs_next_upd_full_value, 577: WiredTiger assertion failed: 'upd->type == 3'. Expression returned false
[1759794889:910410][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1759794889:910419][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910424][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910426][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910427][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910429][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910430][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910432][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910433][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910434][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910436][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910437][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910439][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910440][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910442][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910443][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910445][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910446][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910450][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910451][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910453][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910454][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910456][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910457][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910459][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910460][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910462][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910463][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910465][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910466][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910468][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910469][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910471][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910472][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910474][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910476][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910477][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910479][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910480][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910482][8191:0xffffa9ae0040], t, file:T00001.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1759794889:910483][8191:0xffffa9ae004

logs

format-stress-test-disagg-switch-1 task_log

Logs:

#0  0x0000ffffa82bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffa82bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffa82763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffa8262264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffa8624fe8 [PAC] in __wt_abort (session=session@entry=0x1067bfc8b160) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffa8676228 in __rec_hs_next_upd_full_value (session=session@entry=0x1067bfc8b160, updates=updates@entry=0xffffd7dcc068, older_full_value=0x1067b6674d80, full_value=0x1067be8275c0, updp=updp@entry=0xffffd7dcc058) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_hs.c:577
#5  0x0000ffffa86794a4 in __wti_rec_hs_insert_updates (session=session@entry=0x1067bfc8b160, r=r@entry=0x1067b65f5900, multi=multi@entry=0x1067b6491f80) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_hs.c:1044
#6  0x0000ffffa8667f7c in __rec_hs_wrapup (session=session@entry=0x1067bfc8b160, r=r@entry=0x1067b65f5900) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_write.c:3605
#7  0x0000ffffa866b20c in __rec_write_wrapup (session=session@entry=0x1067bfc8b160, r=r@entry=0x1067b65f5900) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_write.c:3246
#8  0x0000ffffa8671684 in __reconcile (session=session@entry=0x1067bfc8b160, ref=ref@entry=0x1067b4e88be0, salvage=salvage@entry=0x0, flags=flags@entry=21122, page_lockedp=page_lockedp@entry=0xffffd7dcd6ff) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_write.c:381
#9  0x0000ffffa8670388 in __wt_reconcile (session=session@entry=0x1067bfc8b160, ref=ref@entry=0x1067b4e88be0, salvage=salvage@entry=0x0, flags=flags@entry=21122) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/reconcile/rec_write.c:128
#10 0x0000ffffa85e0c4c in __evict_reconcile (session=session@entry=0x1067bfc8b160, ref=ref@entry=0x1067b4e88be0, evict_flags=evict_flags@entry=4) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/evict/evict_page.c:1058
#11 0x0000ffffa85e2b80 in __wt_evict (session=session@entry=0x1067bfc8b160, ref=ref@entry=0x1067b4e88be0, previous_state=previous_state@entry=3 '\003', flags=flags@entry=4) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/evict/evict_page.c:236
#12 0x0000ffffa84ca794 in __wt_page_release_evict (session=session@entry=0x1067bfc8b160, ref=ref@entry=0x1067b4e88be0, flags=flags@entry=0) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/btree/bt_read.c:130
#13 0x0000ffffa848e57c in __wt_page_release (session=session@entry=0x1067bfc8b160, ref=0x1067b4e88be0, flags=flags@entry=0) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/include/btree_inline.h:2165
#14 0x0000ffffa8490774 in __cursor_reset (cbt=cbt@entry=0x1067b87d9e00) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/include/cursor_inline.h:309
#15 0x0000ffffa84913e4 in __wt_btcur_reset (cbt=cbt@entry=0x1067b87d9e00) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/btree/bt_cursor.c:629
#16 0x0000ffffa854e92c in __layered_move_updates (session=session@entry=0x1067bfc8b160, cbt=cbt@entry=0x1067b87d9e00, key=0x1067b998b180, upds=<optimized out>) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_layered.c:1636
#17 0x0000ffffa854ef28 in __layered_copy_ingest_table (session=0x1067bfc8b160, entry=entry@entry=0x1067bfe2a780) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_layered.c:1738
#18 0x0000ffffa8555390 in __layered_drain_ingest_tables (session=session@entry=0x1067bfc86000) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_layered.c:1877
#19 0x0000ffffa855583c in __wti_disagg_conn_config (session=session@entry=0x1067bfc86000, cfg=cfg@entry=0xffffd7dcee58, reconfig=reconfig@entry=true) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_layered.c:1237
#20 0x0000ffffa855e198 in __wti_conn_reconfig (session=session@entry=0x1067bfc86000, cfg=cfg@entry=0xffffd7dcee58) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_reconfig.c:450
#21 0x0000ffffa853ba8c in __conn_reconfigure (wt_conn=<optimized out>, config=<optimized out>) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/src/conn/conn_api.c:1354
#22 0x000000000040baf8 in disagg_switch_roles () at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/test/format/format_disagg.c:54
#23 0x000000000041afd8 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/dfbb107a166315f117503d2497ce05cf/wiredtiger/test/format/t.c:399

logs

Repro Artifacts:

CONFIG

duplicates

WT-15691 (disagg.mode = switch) __layered_copy_ingest_table tombstone assert

Open

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Oct 07 2025 10:10:13 PM UTC
Updated:: Oct 19 2025 03:22:41 AM UTC
Resolved:: Oct 19 2025 03:22:41 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates