Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Disagg CI-blocker, Evergreen
Labels:
- BB-Tools
- CI-blocker

Assigned Teams:

Storage Engines, Storage Engines - Foundations
Sprint:
SE Foundations - 2025-11-21
Story Points:
5
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- 21c36d36aa3a593eb1d760a58d61f7501f8cb4f0(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15691
Count of Linked BFGs (Last 30 days):
65

format-stress-test-disagg-switch-5 on amazon2023-disagg-stress

Host: i-0a91615ed53c83ba8
Project: wiredtiger
Commit: 75defee4
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-5

format-stress-test-disagg-switch-5 task_log

Logs:

[1759555449:585431][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __layered_copy_ingest_table, 1794: WiredTiger assertion failed: 'tombstone != ((void *)0)'. Expression returned false
[1759555449:585454][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1759555449:585464][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585470][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585472][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585473][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585475][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585476][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585478][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585479][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585481][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585482][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585484][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585485][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585487][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585488][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585490][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585491][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585492][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585494][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585497][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585498][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585500][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585501][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585503][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585504][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585506][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585507][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585509][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585510][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585512][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585513][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585515][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585516][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585518][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585519][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585521][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585523][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585524][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585526][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585527][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585529][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curversion_next_single_key, 326: Error at src/cursor/cur_version.c:326: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1759555449:585530][6161:0xffff89a2d040], t, file:T00001.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETUR

logs

format-stress-test-disagg-switch-5 task_log

Logs:

#0  0x0000ffff880bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff880bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff880763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff88062264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff88425208 [PAC] in __wt_abort (session=session@entry=0x70e87fd0fa10) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff8834f828 in __layered_copy_ingest_table (session=0x70e87fd0fa10, entry=entry@entry=0x70e87fe2d4c0) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/conn/conn_layered.c:1794
#5  0x0000ffff88355608 in __layered_drain_ingest_tables (session=session@entry=0x70e87fd08000) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/conn/conn_layered.c:1878
#6  0x0000ffff88355ab4 in __wti_disagg_conn_config (session=session@entry=0x70e87fd08000, cfg=cfg@entry=0xffffd2a65c08, reconfig=reconfig@entry=true) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/conn/conn_layered.c:1238
#7  0x0000ffff8835e4ac in __wti_conn_reconfig (session=session@entry=0x70e87fd08000, cfg=cfg@entry=0xffffd2a65c08) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/conn/conn_reconfig.c:450
#8  0x0000ffff8833ba40 in __conn_reconfigure (wt_conn=<optimized out>, config=<optimized out>) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/src/conn/conn_api.c:1354
#9  0x000000000040baf8 in disagg_switch_roles () at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/test/format/format_disagg.c:54
#10 0x000000000041afd8 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/d53be58c1128c42e393558bc7bc0c1bc/wiredtiger/test/format/t.c:399

logs

Repro Artifacts:

CONFIG

blocks

WT-15671 Assertion upd->type == WT_UPDATE_STANDARD fails in __rec_hs_next_upd_full_value

Closed

duplicates

WT-15669 test/format (disagg.mode=switch) Tombstone is unexpectedly null

Closed

is duplicated by

WT-15699 (disagg.mode=switch) upd->type == 3 assertion failure

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Oct 07 2025 04:38:59 AM UTC
Updated:: Oct 31 2025 01:12:08 AM UTC
Resolved:: Oct 26 2025 10:11:00 PM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates