Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:
- BB-Tools
- CI-blocker

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Foundations - 2025-09-12
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
First Failing Revision:
- 6d5113eef6a8be6694602494354d2e14f16c8d40(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15431
Count of Linked BFGs (Last 30 days):
14

format-stress-data-validation-test-disagg-leader-2 on amazon2023-disagg-stress

Host: i-0f4e8456bc01e48b2
Project: wiredtiger
Commit: 6d5113ee
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-data-validation-test-disagg-leader-2

format-stress-data-validation-test-disagg-leader-2 task_log

Logs:

[1757556358:691134][154714:0xffff08a0eb80], t, file:T00003.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __rec_update_save, 51: WiredTiger assertion failed: 'upd_memsize == 0'. Expression returned false
[1757556358:691155][154714:0xffff08a0eb80], t, file:T00003.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

logs

format-stress-data-validation-test-disagg-leader-2 task_log

Logs:

#0  0x0000ffff9cc0e9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff9cc0e9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff9cbc53a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff9cbb1264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff9c9c7738 [PAC] in __wt_abort (session=session@entry=0x12de3fc88c40) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/os_common/os_abort.c:31
#4  0x0000ffff9c9fb5d8 in __rec_update_save (session=session@entry=0x12de3fc88c40, r=r@entry=0x12de3558cc80, ins=ins@entry=0x0, rip=rip@entry=0x12de3bfe3b80, onpage_upd=onpage_upd@entry=0x0, tombstone=0x12de3efbed40, tw=tw@entry=0xffff08a0c358, supd_restore=supd_restore@entry=false, upd_memsize=96) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/reconcile/rec_visibility.c:51
#5  0x0000ffff9c9fd25c in __wti_rec_upd_select (session=session@entry=0x12de3fc88c40, r=r@entry=0x12de3558cc80, ins=ins@entry=0x0, rip=rip@entry=0x12de3bfe3b80, vpack=vpack@entry=0xffff08a0c3c8, upd_select=upd_select@entry=0xffff08a0c348) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/reconcile/rec_visibility.c:1351
#6  0x0000ffff9c9f4d30 in __wti_rec_row_leaf (session=session@entry=0x12de3fc88c40, r=r@entry=0x12de3558cc80, pageref=pageref@entry=0x12de3e6a92c0, salvage=salvage@entry=0x0) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/reconcile/rec_row.c:1018
#7  0x0000ffff9ca07f9c in __reconcile (session=session@entry=0x12de3fc88c40, ref=ref@entry=0x12de3e6a92c0, salvage=salvage@entry=0x0, flags=flags@entry=132, page_lockedp=page_lockedp@entry=0xffff08a0cd9f) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/reconcile/rec_write.c:305
#8  0x0000ffff9ca0726c in __wt_reconcile (session=session@entry=0x12de3fc88c40, ref=0x12de3e6a92c0, salvage=salvage@entry=0x0, flags=flags@entry=132) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/reconcile/rec_write.c:124
#9  0x0000ffff9c8cb698 in __wt_sync_file (session=session@entry=0x12de3fc88c40, syncop=syncop@entry=WT_SYNC_CHECKPOINT) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/btree/bt_sync.c:364
#10 0x0000ffff9c8fc9f0 in __checkpoint_tree (session=session@entry=0x12de3fc88c40, is_checkpoint=is_checkpoint@entry=true, cfg=cfg@entry=0xffff08a0ddd8) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:2560
#11 0x0000ffff9c8fce84 in __checkpoint_tree_helper (session=0x12de3fc88c40, cfg=0xffff08a0ddd8) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:2698
#12 0x0000ffff9c8f72cc in __checkpoint_apply_to_dhandles (session=session@entry=0x12de3fc88c40, cfg=cfg@entry=0xffff08a0ddd8, op=op@entry=0xffff9c8fce00 <__checkpoint_tree_helper>) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:340
#13 0x0000ffff9c8fd848 in __checkpoint_db_internal (session=session@entry=0x12de3fc88c40, cfg=cfg@entry=0xffff08a0ddd8) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:1343
#14 0x0000ffff9c8fe770 in __checkpoint_db_wrapper (session=session@entry=0x12de3fc88c40, cfg=cfg@entry=0xffff08a0ddd8) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:1734
#15 0x0000ffff9c8fea30 in __wt_checkpoint_db (session=session@entry=0x12de3fc88c40, cfg=cfg@entry=0xffff08a0ddd8, waiting=waiting@entry=true) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/checkpoint/checkpoint_txn.c:1813
#16 0x0000ffff9ca337a0 in __session_checkpoint (wt_session=0x12de3fc88c40, config=<optimized out>) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/session/session_api.c:2363
#17 0x0000000000405a88 in checkpoint (arg=<optimized out>) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/test/format/checkpoint.c:147
#18 0x0000ffff9cc0cd78 in start_thread () from /lib64/libc.so.6
#19 0x0000ffff9cc79ddc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is caused by

WT-15389 Saved update lists with deletes are lost when building a delta

Closed

is related to

WT-15438 Failed to promote a node to primary due to reconfigure returning WT_NOTFOUND

Closed

WT-15210 Change eviction to scrub eviction when the cache usage is less than eviction target

Open

WT-15428 Bypass materialisation frontier check on standby for eviction

Closed

WT-15424 Add extra logging for disaggregated storage

Closed

WT-15389 Saved update lists with deletes are lost when building a delta

Closed

related to

WT-15438 Failed to promote a node to primary due to reconfigure returning WT_NOTFOUND

Closed

WT-14981 Fix disagg hook to handle multithreading in tests

Closed

WT-15247 Unexpected keys observed after connection reopen for disagg

Closed

WT-15300 Set backlink LSN in __wt_disagg_put_meta

Closed

WT-14911 Verify that the ingest table is always empty for primary

Closed

WT-15198 Data Validation for Leader/Follower Mode in Disagg (test/format)

Closed

WT-15240 Enable test_timestamp*.py for disagg in Evergreen

Closed

WT-15412 Add a table id as part of the WT_BLOCK_DISAGG struct

Closed

WT-15428 Bypass materialisation frontier check on standby for eviction

Closed

WT-15424 Add extra logging for disaggregated storage

Closed

(1 is related to, 10 related to)

Assignee:: Chenhao Qu
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Sep 11 2025 04:23:32 AM UTC
Updated:: Sep 12 2025 07:34:50 AM UTC
Resolved:: Sep 11 2025 09:08:38 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates