Loading...

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Transactions
Sprint:
SE Transactions - 2026-02-27
Story Points:
5
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-stress-nonstandalone
Failing Tasks:
- format-stress-test-1
First Failing Revision:
- a48e8711c35fb4f903de1333913080c1c503662a(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16652
Count of Linked BFGs (Last 30 days):
1

format-stress-test-1 on amazon2023-stress-nonstandalone

Host: i-0346c68bceacec2ac
Project: wiredtiger
Commit: a48e8711
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-1

format-stress-test-1 task_log

Logs:

    [1770777355:437535][134729:0xffff82b1dbc0], t, file:T00001.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_block_compact_start, 24: Foreground compaction already happening on data handle T00001.wt by session 47: Device or resource busy
    [1770777437:119821][134729:0xffff68e8dbc0], t, file:T00002.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_block_compact_start, 24: Background compaction already happening on data handle T00002.wt by session 16: Device or resource busy
    [1770777460:140075][134729:0xffff68e8dbc0], t, file:T00002.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_block_compact_start, 24: Background compaction already happening on data handle T00002.wt by session 16: Device or resource busy
    [1770777966:339820][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_DEFAULT][ERROR]: __wt_evict_file, 119: WiredTiger assertion failed: '((((dhandle)->flags) & (0x001u)) != 0) || (__wt_atomic_load_uint32_relaxed(&((((WT_CONNECTION_IMPL *)((WT_SESSION_IMPL *)(session))->iface.connection))->flags_atomic)) & (uint32_t)(0x00004u)) || __wt_page_can_evict(session, ref, ((void *)0))'. Page should be evictable during discard
    [1770777966:339850][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
    [1770777966:339860][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339864][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339866][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339868][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339870][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339872][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339874][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339875][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339877][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339879][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339881][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339891][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339893][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339895][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339896][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339898][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339900][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339901][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339903][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
    [1770777966:339904][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339906][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339908][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339909][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339911][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339913][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339914][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339916][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339920][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339922][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339923][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339925][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339927][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339928][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
    [1770777966:339930][134729:0xffff6c0ddbc0], t, file:T00002.wt, WT_SESSION.alter: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found

logs

format-stress-test-1 task_log

Logs:

test/format run configuration highlights

logs

format-stress-test-1 task_log

Logs:

#0  0x0000ffff8bcbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff8bcbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff8bc763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff8bc62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8c0365bc [PAC] in __wt_abort (session=session@entry=0x10af3feb7d50) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff8bfe1f14 in __wt_evict_file (session=session@entry=0x10af3feb7d50, syncop=syncop@entry=WT_SYNC_DISCARD) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/evict/evict_file.c:119
#5  0x0000ffff8bf3074c in __wt_checkpoint_close (session=session@entry=0x10af3feb7d50, final=final@entry=false) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/checkpoint/checkpoint_txn.c:2981
#6  0x0000ffff8bf54874 in __wt_conn_dhandle_close (session=session@entry=0x10af3feb7d50, final=final@entry=false, mark_dead=mark_dead@entry=false, check_visibility=check_visibility@entry=false) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/conn/conn_dhandle.c:471
#7  0x0000ffff8bf54ff0 in __conn_dhandle_close_one (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af1caf9730 "file:T00002.wt", checkpoint=checkpoint@entry=0x0, removed=removed@entry=false, mark_dead=mark_dead@entry=false, check_visibility=check_visibility@entry=false) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/conn/conn_dhandle.c:856
#8  0x0000ffff8bf55bec in __wt_conn_dhandle_close_all (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af1caf9730 "file:T00002.wt", removed=removed@entry=false, mark_dead=mark_dead@entry=false, check_visibility=check_visibility@entry=false) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/conn/conn_dhandle.c:896
#9  0x0000ffff8c0a3660 in __wti_execute_handle_operation (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af1caf9730 "file:T00002.wt", file_func=file_func@entry=0xffff8c08af5c <__alter_file>, cfg=cfg@entry=0xffff6c0dce58, open_flags=open_flags@entry=288) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_worker.c:26
#10 0x0000ffff8c08ba2c in __schema_alter (session=session@entry=0x10af3feb7d50, uri=0x10af1caf9730 "file:T00002.wt", newcfg=newcfg@entry=0xffff6c0dce58) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_alter.c:419
#11 0x0000ffff8c08bbc0 in __alter_tree (session=session@entry=0x10af3feb7d50, name=0x10af3fc02bb0 "colgroup:T00002", newcfg=newcfg@entry=0xffff6c0dce58) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_alter.c:312
#12 0x0000ffff8c08c07c in __alter_table (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af3fe32004 "table:T00002", newcfg=newcfg@entry=0xffff6c0dce58, exclusive_refreshed=exclusive_refreshed@entry=true) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_alter.c:367
#13 0x0000ffff8c08ba8c in __schema_alter (session=0x10af3feb7d50, uri=uri@entry=0x10af3fe32004 "table:T00002", newcfg=newcfg@entry=0xffff6c0dce58) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_alter.c:425
#14 0x0000ffff8c08c400 in __wt_schema_alter (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af3fe32004 "table:T00002", newcfg=newcfg@entry=0xffff6c0dce58) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/schema/schema_alter.c:454
#15 0x0000ffff8c0adc6c in __session_alter_internal (session=session@entry=0x10af3feb7d50, uri=uri@entry=0x10af3fe32004 "table:T00002", cfg=cfg@entry=0xffff6c0dce58) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/session/session_api.c:970
#16 0x0000ffff8c0ae1ac in __session_alter (wt_session=0x10af3feb7d50, uri=0x10af3fe32004 "table:T00002", config=<optimized out>) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/src/session/session_api.c:1044
#17 0x000000000040317c in alter (arg=<optimized out>) at /data/mci/b1da7dbb6c7e2d1526165a0b3af7fe79/wiredtiger/test/format/alter.c:73
#18 0x0000ffff8bcbdb78 in start_thread () from /lib64/libc.so.6
#19 0x0000ffff8bd2acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

related to

WT-15298 __wt_evict_file, 111: WiredTiger assertion failed

Open

Details

Description

Attachments

Issue Links

Activity

People

Dates