Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 9.0.0-rc0
Affects Version/s: None
Component/s: Verify
Security Level: Public (Available to anyone on the web)
Labels:
- BB-Tools
- lc_bulk_04_29_26

Assigned Teams:

Storage Engines - Persistence
Total Hours with Assigned Team:
1,706.053
Epic Link:
Validation improvements in disagg
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- 27ea5d1ac99008cbe0acb2b3c99a8ddf6c0a2140(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16703
Count of Linked BFGs (Last 30 days):
2

format-stress-test-disagg-leader-pull-request on ubuntu2004-ubsan

Host: i-0762f06897f6e5ed0
Project: wiredtiger
Commit: 27ea5d1a
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-pull-request

format-stress-test-disagg-leader-pull-request task_log

Logs:

    [1771301831:056643][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: int __schema_layered_worker_verify(WT_SESSION_IMPL *, const char *, int (*)(WT_SESSION_IMPL *, const char **), int (*)(WT_SESSION_IMPL *, const char *, _Bool *), const char **, uint32_t), 132: WiredTiger assertion failed: 'ingest_ret != 16'. Verify: file:T00002.wt_ingest ingest table on leader cannot be verified. Ingest contains dirty content or open cursors, which is an invalid state.
    [1771301831:056684][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 29: aborting WiredTiger library
    [1771301831:056695][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_prev(WT_CURSOR_BTREE *, _Bool), 706: Error at src/btree/bt_curprev.c:706: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1771301831:056707][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_prev(WT_CURSOR *), 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
    [1771301831:056713][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 879: Error at src/cursor/cur_hs.c:879: "ret" failed: WT_NOTFOUND: item not found
    [1771301831:056719][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1771301831:056725][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_session_get_dhandle(WT_SESSION_IMPL *, const char *, const char *, const char **, uint32_t), 939: Error at src/session/session_dhandle.c:939: "__wt_session_lock_dhandle(session, flags, &is_dead)" failed: Device or resource busy
    [1771301831:056736][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __conn_dhandle_close_one(WT_SESSION_IMPL *, const char *, const char *, _Bool, _Bool, _Bool), 846: Error at src/conn/conn_dhandle.c:846: "__wt_session_get_dhandle( session, uri, checkpoint, NULL, WT_DHANDLE_EXCLUSIVE | WT_DHANDLE_LOCK_ONLY)" failed: Device or resource busy
    [1771301831:056743][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_conn_dhandle_close_all(WT_SESSION_IMPL *, const char *, _Bool, _Bool, _Bool), 896: Error at src/conn/conn_dhandle.c:896: "__conn_dhandle_close_one(session, uri, NULL, removed, mark_dead, check_visibility)" failed: Device or resource busy
    [1771301831:056749][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __wti_execute_handle_operation(WT_SESSION_IMPL *, const char *, int (*)(WT_SESSION_IMPL *, const char **), const char **, uint32_t), 28: Error at src/schema/schema_worker.c:28: "ret" failed: Device or resource busy
    [1771301831:056755][13120:0x7f0b2810b700], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_schema_worker(WT_SESSION_IMPL *, const char *, int (*)(WT_SESSION_IMPL *, const char **), int (*)(WT_SESSION_IMPL *, const char *, _Bool *), const char **, uint32_t), 191: Error at src/schema/schema_worker.c:191: "__wti_execute_handle_operation(session, uri, file_func, cfg, open_flags)" failed: Device or resource busy

logs

format-stress-test-disagg-leader-pull-request task_log

Logs:

test/format run configuration highlights

logs

format-stress-test-disagg-leader-pull-request task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f0b28132859 in __GI_abort () at abort.c:79
#2  0x00007f0b29dfe615 in __wt_abort (session=0x55c9bd23f6f0) at /data/mci/22b839681c011b3146f147be6bebd2ff/wiredtiger/src/os_common/os_abort.c:32
#3  0x00007f0b2a0fc155 in __schema_layered_worker_verify (session=0x55c9bd23f6f0, uri=0x55c9bdce1fd0 "layered:T00002", file_func=0x7f0b295bc908 <__wt_verify>, name_func=0x0, cfg=0x7ffc41d83220, open_flags=16777248) at /data/mci/22b839681c011b3146f147be6bebd2ff/wiredtiger/src/schema/schema_worker.c:128
#4  0x00007f0b2a0fab31 in __wt_schema_worker (session=0x55c9bd23f6f0, uri=0x55c9bdce1fd0 "layered:T00002", file_func=0x7f0b295bc908 <__wt_verify>, name_func=0x0, cfg=0x7ffc41d83220, open_flags=16777248) at /data/mci/22b839681c011b3146f147be6bebd2ff/wiredtiger/src/schema/schema_worker.c:247
#5  0x00007f0b2a0fa587 in __wt_schema_worker (session=0x55c9bd23f6f0, uri=0x55c9bcfd9414 "table:T00002", file_func=0x7f0b295bc908 <__wt_verify>, name_func=0x0, cfg=0x7ffc41d83220, open_flags=16777248) at /data/mci/22b839681c011b3146f147be6bebd2ff/wiredtiger/src/schema/schema_worker.c:225
#6  0x00007f0b2a14a6da in __session_verify (wt_session=0x55c9bd23f6f0, uri=0x55c9bcfd9414 "table:T00002", config=0x55c9ba87c76b "strict") at /data/mci/22b839681c011b3146f147be6bebd2ff/wiredtiger/src/session/session_api.c:1817
#7  0x000055c9ba83034d in table_verify ()
#8  0x000055c9ba8311fd in tables_apply ()
#9  0x000055c9ba830965 in wts_verify ()
#10 0x000055c9ba82ce33 in main ()

logs

Repro Artifacts:

CONFIG

is duplicated by

WT-17071 Dirty content in ingest tables during verification

Closed

WT-16938 failed: format-stress-test-disagg-leader-1 on ubuntu2004-stress-nonstandalone [wiredtiger @ 67e89867]

Closed

is related to

WT-14734 Layered table lock potentially not being held long enough

Backlog

WT-16962 Add diagnostic post-condition check to ensure ingest table is empty after truncation

Closed

WT-16695 Layered truncate: fix assert and re-enable test

Closed

related to

WT-16974 Prevent sweep server from closing ingest and layered dhandles

Closed

WT-17071 Dirty content in ingest tables during verification

Closed

WT-16837 Investigate whether the stat log server should process ingest tables on leader

Open

WT-16982 Provide long term solution layered dhandles and ingest tables

Backlog

WT-16531 test/format (disagg.mode=switch) data mismatch (part 2)

Closed

(5 related to)

Assignee:: Etienne Petrel
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 10 Start watching this issue

Created:: Feb 17 2026 10:24:39 PM UTC
Updated:: May 04 2026 11:36:42 PM UTC
Resolved:: Apr 13 2026 11:53:42 PM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates