Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Metadata
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Foundations, Storage Engines - Transactions
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- 4a5646cb24fc0464a0b604905523cf0064fb94c3(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-17112
Count of Linked BFGs (Last 30 days):
9

format-stress-test-disagg-leader-data-validation-1 on ubuntu2004-stress-nonstandalone

Host: i-0eb2952a1872466b3
Project: wiredtiger
Commit: 4a5646cb
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-data-validation-1

format-stress-test-disagg-leader-data-validation-1 task_log

Logs:

    [1775533945:675409][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_btree_open, 213: WiredTiger assertion failed: '!(((session->lock_flags) & (0x0100u)) != 0) || (((session->lock_flags) & (0x0001u)) != 0)'. deadlock
    [1775533945:675444][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
    [1775533945:675463][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675473][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675478][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675482][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675487][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675491][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675495][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675500][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_session_get_dhandle, 939: Error at src/session/session_dhandle.c:939: "__wt_session_lock_dhandle(session, flags, &is_dead)" failed: Device or resource busy
    [1775533945:675509][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675513][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675518][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675522][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675527][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675531][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675535][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675546][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675551][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675555][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675559][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675563][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675567][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675572][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675576][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675581][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675585][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675589][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675593][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675598][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675602][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675606][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675610][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675617][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675622][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675627][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675631][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675635][11704:0x7f118b612700], t, file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1775533945:675639][11704:0x7f118b612700], t, file:Wi

logs

format-stress-test-disagg-leader-data-validation-1 task_log

Logs:

test/format run configuration highlights

logs

format-stress-test-disagg-leader-data-validation-1 task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f118c3e6859 in __GI_abort () at abort.c:79
#2  0x00007f118c9c4a7b in __wt_abort (session=session@entry=0x52bc7fc94f90) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/os_common/os_abort.c:32
#3  0x00007f118c8207e8 in __wt_btree_open (session=session@entry=0x52bc7fc94f90, op_cfg=op_cfg@entry=0x0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/btree/bt_handle.c:213
#4  0x00007f118c8d12b7 in __wt_conn_dhandle_open (session=session@entry=0x52bc7fc94f90, cfg=cfg@entry=0x0, flags=flags@entry=0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/conn/conn_dhandle.c:647
#5  0x00007f118ca5b7df in __wt_session_get_dhandle (session=0x52bc7fc94f90, uri=0x52bc7fe0f480 "file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1", checkpoint=0x0, cfg=0x0, flags=0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/session/session_dhandle.c:986
#6  0x00007f118ca5bda6 in __wt_session_get_dhandle (session=session@entry=0x52bc7fc94f90, uri=0x52bc7fe0f480 "file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.1", checkpoint=0x0, cfg=cfg@entry=0x0, flags=flags@entry=0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/session/session_dhandle.c:980
#7  0x00007f118c8cf320 in __conn_btree_apply_internal (session=session@entry=0x52bc7fc94f90, dhandle=dhandle@entry=0x52bc7f7fb400, file_func=file_func@entry=0x7f118c8ef25b <__statlog_apply>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/conn/conn_dhandle.c:744
#8  0x00007f118c8d18d4 in __wt_conn_btree_apply (session=session@entry=0x52bc7fc94f90, uri=uri@entry=0x0, file_func=file_func@entry=0x7f118c8ef25b <__statlog_apply>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x0) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/conn/conn_dhandle.c:812
#9  0x00007f118c8eedcf in __statlog_log_one (session=session@entry=0x52bc7fc94f90, path=path@entry=0x7f118b6109b0, tmp=tmp@entry=0x7f118b610980) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/conn/conn_stat.c:460
#10 0x00007f118c8eef66 in __statlog_server (arg=0x52bc7fc94f90) at /data/mci/b693869d38d0e1a39cfe5582f9116cfd/wiredtiger/src/conn/conn_stat.c:545
#11 0x00007f118c73a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007f118c4e3353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

logs

Repro Artifacts:

CONFIG

duplicates

WT-17103 Deadlock assertion when statlog and sweep race on shared metadata on leader

Closed

Assignee:: Sid Mahajan
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Apr 08 2026 03:42:17 AM UTC
Updated:: Apr 14 2026 10:18:53 AM UTC
Resolved:: Apr 12 2026 06:19:29 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates