Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Disagg CI-blocker
Labels:
- BB-Tools
- CI-blocker

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Transactions - 2025-10-24, SE Transactions - 2025-11-07
Story Points:
1
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- 1fbb6e10423a76ed1336180c8eceb11f0679d476(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15716
Count of Linked BFGs (Last 30 days):
63

format-stress-test-disagg-switch-1 on amazon2023-disagg-stress

Host: i-080453b52bf46092e
Project: wiredtiger
Commit: 9a827073
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-1

format-stress-test-disagg-switch-1 task_log

Logs:

[1759967347:458673][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __reconcile, 341: WiredTiger assertion failed: '!((page->type) == 7 ? (((((((WT_BTREE *)(session)->dhandle->handle))->flags) & (0x0004000u)) != 0) && ((((&((WT_CONNECTION_IMPL *)((WT_SESSION_IMPL *)(session))->iface.connection)->page_delta)->flags) & (0x4u)) != 0)) : (((((((WT_BTREE *)(session)->dhandle->handle))->flags) & (0x0004000u)) != 0) && ((((&((WT_CONNECTION_IMPL *)((WT_SESSION_IMPL *)(session))->iface.connection)->page_delta)->flags) & (0x2u)) != 0))) || ((((r)->flags) & (0x0040u)) != 0) || page->disagg_info->block_meta.page_id == 0'. Expression returned false
[1759967347:458702][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1759967347:458708][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 205: Error at src/evict/evict_page.c:205: "__evict_exclusive(session, ref)" failed: Device or resource busy
[1759967347:458717][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1759967347:458719][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1759967347:458721][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1759967347:458723][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1759967347:458724][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1759967347:458726][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1759967347:458730][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1759967347:458731][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1759967347:458733][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 205: Error at src/evict/evict_page.c:205: "__evict_exclusive(session, ref)" failed: Device or resource busy
[1759967347:458735][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 205: Error at src/evict/evict_page.c:205: "__evict_exclusive(session, ref)" failed: Device or resource busy
[1759967347:458736][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 205: Error at src/evict/evict_page.c:205: "__evict_exclusive(session, ref)" failed: Device or resource busy
[1759967347:458738][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 205: Error at src/evict/evict_page.c:205: "__evict_exclusive(session, ref)" failed: Device or resource busy
[1759967347:458740][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_casv64(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1759967347:458742][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_reconcile, 1068: Error at src/evict/evict_page.c:1068: "ret" failed: Device or resource busy
[1759967347:458744][4282:0xffff0e41dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 236: Error at src/evict/evict_page.c:236: "__evict_reconcile(session, ref, flags)" failed: Device or resource busy

logs

format-stress-test-disagg-switch-1 task_log

Logs:

#0  0x0000ffffa18bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffa18bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffa18763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffa1862264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffa1c24d18 [PAC] in __wt_abort (session=session@entry=0x30dcffc98878) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffa1c71324 in __reconcile (session=session@entry=0x30dcffc98878, ref=ref@entry=0x30dcf53826e0, salvage=salvage@entry=0x0, flags=flags@entry=4746, page_lockedp=page_lockedp@entry=0xffff0e41afff) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/reconcile/rec_write.c:341
#5  0x0000ffffa1c700f8 in __wt_reconcile (session=session@entry=0x30dcffc98878, ref=ref@entry=0x30dcf53826e0, salvage=salvage@entry=0x0, flags=flags@entry=4746) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/reconcile/rec_write.c:128
#6  0x0000ffffa1be097c in __evict_reconcile (session=session@entry=0x30dcffc98878, ref=ref@entry=0x30dcf53826e0, evict_flags=evict_flags@entry=4) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/evict/evict_page.c:1058
#7  0x0000ffffa1be28b0 in __wt_evict (session=session@entry=0x30dcffc98878, ref=ref@entry=0x30dcf53826e0, previous_state=previous_state@entry=3 '\003', flags=flags@entry=4) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/evict/evict_page.c:236
#8  0x0000ffffa1aca574 in __wt_page_release_evict (session=session@entry=0x30dcffc98878, ref=0x30dcf53826e0, flags=flags@entry=0) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/btree/bt_read.c:130
#9  0x0000ffffa1ae98dc in __wt_sync_file (session=session@entry=0x30dcffc98878, syncop=syncop@entry=WT_SYNC_CHECKPOINT) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/btree/bt_sync.c:347
#10 0x0000ffffa1b23e8c in __checkpoint_tree (session=session@entry=0x30dcffc98878, is_checkpoint=is_checkpoint@entry=true, cfg=cfg@entry=0xffff0e41ce18) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:2568
#11 0x0000ffffa1b24534 in __checkpoint_tree_helper (session=0x30dcffc98878, cfg=0xffff0e41ce18) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:2706
#12 0x0000ffffa1b1d1c4 in __checkpoint_apply_to_dhandles (session=session@entry=0x30dcffc98878, cfg=cfg@entry=0xffff0e41ce18, op=op@entry=0xffffa1b244b0 <__checkpoint_tree_helper>) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:340
#13 0x0000ffffa1b2513c in __checkpoint_db_internal (session=session@entry=0x30dcffc98878, cfg=cfg@entry=0xffff0e41ce18) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:1348
#14 0x0000ffffa1b26484 in __checkpoint_db_wrapper (session=session@entry=0x30dcffc98878, cfg=cfg@entry=0xffff0e41ce18) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:1742
#15 0x0000ffffa1b2685c in __wt_checkpoint_db (session=session@entry=0x30dcffc98878, cfg=cfg@entry=0xffff0e41ce18, waiting=waiting@entry=true) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/checkpoint/checkpoint_txn.c:1821
#16 0x0000ffffa1ca75a4 in __session_checkpoint (wt_session=0x30dcffc98878, config=<optimized out>) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/session/session_api.c:2433
#17 0x0000000000405bf0 in checkpoint (arg=<optimized out>) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/test/format/checkpoint.c:147
#18 0x0000ffffa18bdb78 in start_thread () from /lib64/libc.so.6
#19 0x0000ffffa192acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is related to

WT-15877 Tune SQLite3 to improve performace

Closed

WT-15726 disagg - TimestampMonitor hit a null pointer on startup

Closed

WT-15748 Add stat to count how many ingest btrees and stable btrees are skipped by eviction server

Closed

WT-15878 Allow verify to return errors in salvage

Closed

WT-15826 test/format (disagg.mode=leader, PALM) OOO keys detected

Closed

related to

WT-15730 test/format (disagg.mode=leader) assertion __wti_rec_row_int - Not propagating the original fast-truncate information

Closed

WT-14700 Discard root pages for old checkpoints

Closed

WT-15936 (disagg.mode=leader) test/format __reconcile (rec_write.c:341) assertion failure

Closed

(3 related to)

Assignee:: Chenhao Qu
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 6 Start watching this issue

Created:: Oct 09 2025 01:58:32 AM UTC
Updated:: Nov 02 2025 11:47:13 PM UTC
Resolved:: Oct 29 2025 01:39:39 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates