Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Foundations
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
First Failing Revision:
- 2751433463b1e8c70d70c559da69ad4a652b455b(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16693
Count of Linked BFGs (Last 30 days):
8

format-stress-test-disagg-switch-1 on amazon2023-disagg-stress

Host: i-0f13b93cc5fc3c161
Project: wiredtiger
Commit: 27514334
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-1

format-stress-test-disagg-switch-1 task_log

Logs:

[1771278544:904971][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __checkpoint_update_disagg_database_size, 904: WiredTiger assertion failed: 'db >= (uint64_t)(-delta)'. Expression returned false
[1771278544:904989][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1771278544:905000][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905006][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905008][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905010][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905011][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905013][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905015][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905017][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905018][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905020][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905022][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905024][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905025][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905027][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905029][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905031][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905034][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905036][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905038][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905040][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905041][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905043][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905045][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905047][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905048][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905050][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905052][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905054][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905055][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905057][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905059][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905061][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905063][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905065][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905067][4479:0xfff4653cdbc0], t, file:WiredTigerShared.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 751: Error at src/cursor/cur_hs.c:751: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1771278544:905069][4479:0xfff4653cdbc0], t, file:Wi

logs

format-stress-test-disagg-switch-1 task_log

Logs:

#0  0x0000ffffb92c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffb92c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffb927b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffb9262224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffb9635570 [PAC] in __wt_abort (session=session@entry=0x5303bfc9dc40) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffb952ba14 in __checkpoint_update_disagg_database_size (session=session@entry=0x5303bfc9dc40) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/checkpoint/checkpoint_txn.c:904
#5  0x0000ffffb952eb58 in __checkpoint_db_internal (session=session@entry=0x5303bfc9dc40, cfg=cfg@entry=0xfff4653cce18) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/checkpoint/checkpoint_txn.c:1782
#6  0x0000ffffb952f280 in __checkpoint_db_wrapper (session=session@entry=0x5303bfc9dc40, cfg=cfg@entry=0xfff4653cce18) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/checkpoint/checkpoint_txn.c:1932
#7  0x0000ffffb952f660 in __wt_checkpoint_db (session=session@entry=0x5303bfc9dc40, cfg=cfg@entry=0xfff4653cce18, waiting=waiting@entry=true) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/checkpoint/checkpoint_txn.c:2011
#8  0x0000ffffb96b7a4c in __session_checkpoint (wt_session=0x5303bfc9dc40, config=<optimized out>) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/src/session/session_api.c:2443
#9  0x0000000000405f08 in checkpoint (arg=<optimized out>) at /data/mci/02ae562903512c5e8244f29d4caf3722/wiredtiger/test/format/checkpoint.c:147
#10 0x0000ffffb92c2834 in start_thread () from /lib64/libc.so.6
#11 0x0000ffffb9266e5c [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

Assignee:: Sid Mahajan
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Feb 16 2026 11:33:21 PM UTC
Updated:: Feb 18 2026 02:16:57 PM UTC
Resolved:: Feb 17 2026 02:40:38 AM UTC

Details

Description

Attachments

Activity

People

Dates