Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:
- BB-Tools
- CI-blocker

Assigned Teams:

Storage Engines, Storage Engines - Persistence
Sprint:
SE Transactions - 2025-09-26
Story Points:
5
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-asan-stress
- amazon2023-disagg-stress
Failing Tasks:
First Failing Revision:
- 61d4086168cdf12f1c2863f6d88f2b5c8d3ea8a6(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15450
Count of Linked BFGs (Last 30 days):
49

format-stress-data-validation-test-disagg-leader-2 on amazon2023-disagg-stress

Host: i-0e7086d4fb31b2ca5
Project: wiredtiger
Commit: 61d40861
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-data-validation-test-disagg-leader-2

format-stress-data-validation-test-disagg-leader-2 task_log

Logs:

[1757590416:971579][48838:0xffffb1aae040], t, file:T00003.wt_stable, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __wt_cell_unpack_addr, 1399: WiredTiger assertion failed: 'ret == 0'. Expression returned false
[1757590416:971600][48838:0xffffb1aae040], t, file:T00003.wt_stable, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

logs

format-stress-data-validation-test-disagg-leader-2 task_log

Logs:

#0  0x0000ffffb02bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffb02bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffb02763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffb0262264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffb05c7bdc [PAC] in __wt_abort (session=session@entry=0x704cbf805e80) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/os_common/os_abort.c:31
#4  0x0000ffffb04d3a30 in __wt_cell_unpack_addr (session=session@entry=0x704cbf805e80, dsk=0x704cb5640000, cell=<optimized out>, unpack_addr=unpack_addr@entry=0xfffff5674bb0) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/include/cell_inline.h:1399
#5  0x0000ffffb04d736c in __verify_tree (session=session@entry=0x704cbf805e80, ref=ref@entry=0x704cba8ec280, addr_unpack=addr_unpack@entry=0xfffff56750d0, vs=vs@entry=0xfffff5675bf8) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/btree/bt_vrfy.c:851
#6  0x0000ffffb04d71c8 in __verify_tree (session=session@entry=0x704cbf805e80, ref=ref@entry=0x704cbffd6fa0, addr_unpack=addr_unpack@entry=0xfffff56755f0, vs=vs@entry=0xfffff5675bf8) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/btree/bt_vrfy.c:874
#7  0x0000ffffb04d71c8 in __verify_tree (session=session@entry=0x704cbf805e80, ref=ref@entry=0x704cbc9ff998, addr_unpack=addr_unpack@entry=0xfffff5677240, vs=vs@entry=0xfffff5675bf8) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/btree/bt_vrfy.c:874
#8  0x0000ffffb04d7a38 in __wt_verify (session=0x704cbf805e80, cfg=<optimized out>) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/btree/bt_vrfy.c:311
#9  0x0000ffffb06212e4 in __wti_execute_handle_operation (session=session@entry=0x704cbf805e80, uri=uri@entry=0x704cbfe45ae0 "file:T00003.wt_stable", file_func=file_func@entry=0xffffb04d7450 <__wt_verify>, cfg=cfg@entry=0xfffff5677d58, open_flags=open_flags@entry=16777248) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/schema/schema_worker.c:32
#10 0x0000ffffb06214e0 in __wt_schema_worker (session=session@entry=0x704cbf805e80, uri=uri@entry=0x704cbfe45ae0 "file:T00003.wt_stable", file_func=file_func@entry=0xffffb04d7450 <__wt_verify>, name_func=name_func@entry=0x0, cfg=cfg@entry=0xfffff5677d58, open_flags=open_flags@entry=16777248) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/schema/schema_worker.c:194
#11 0x0000ffffb06219cc in __schema_layered_worker_verify (session=session@entry=0x704cbf805e80, uri=uri@entry=0x704cbfe48d10 "layered:T00003", file_func=file_func@entry=0xffffb04d7450 <__wt_verify>, name_func=name_func@entry=0x0, cfg=cfg@entry=0xfffff5677d58, open_flags=open_flags@entry=16777248) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/schema/schema_worker.c:111
#12 0x0000ffffb0621828 in __wt_schema_worker (session=0x704cbf805e80, uri=0x704cbfe48d10 "layered:T00003", file_func=0xffffb04d7450 <__wt_verify>, name_func=0x0, cfg=0xfffff5677d58, open_flags=16777248) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/schema/schema_worker.c:250
#13 0x0000ffffb0621760 in __wt_schema_worker (session=session@entry=0x704cbf805e80, uri=uri@entry=0x704cbfc1c004 "table:T00003", file_func=0xffffb04d7450 <__wt_verify>, name_func=name_func@entry=0x0, cfg=cfg@entry=0xfffff5677d58, open_flags=16777248) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/schema/schema_worker.c:228
#14 0x0000ffffb06263a0 in __session_verify (wt_session=0x704cbf805e80, uri=0x704cbfc1c004 "table:T00003", config=<optimized out>) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/src/session/session_api.c:1742
#15 0x000000000041ab5c in table_verify (table=0x704cbfc1c000, arg=0x704cbfc1e000) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/test/format/verify.c:49
#16 0x000000000041aadc in tables_apply (func=func@entry=0x41aafc <table_verify>, arg=arg@entry=0x704cbfc1e000) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/test/format/format_inline.h:186
#17 0x000000000041c460 in wts_verify (conn=0x704cbfc1e000, mirror_check=mirror_check@entry=true) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/test/format/verify.c:447
#18 0x000000000041a228 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/f70f8a583b137cd42f803c2ddd54ffa5/wiredtiger/test/format/t.c:414

logs

Repro Artifacts:

CONFIG

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

T00003.internal_page_dump.txt
81 kB
Sep 16 2025 02:51:12 PM UTC
WT-15450-reproducer.diff
8 kB
Sep 18 2025 04:27:30 AM UTC

is related to

WT-15575 Prevent the follower from writing checkpoint metadata

Closed

WT-14885 Rewrite pages with delta directly to a full page

Closed

WT-15563 Investigate making cache tolerant to change app step-wise eviction to incremental eviction

Closed

WT-15568 Add the ability to dump the error log without the connection object

Closed

WT-15576 Use EIO to indicate I/O errors in disagg

Closed

WT-15410 test/format (disagg.mode=leader) Delta chain validation failed

Closed

WT-15455 Don't skip prepared update pages during cursor walk

Closed

WT-15550 layered38, 32 data mismatch error

Closed

WT-15566 Remove table_id parameter from palm_handle_get_page_ids

Closed

related to

WT-15215 Investigate all related test/format (disagg) detects out-of-order keys

Closed

WT-15539 test/format (disagg) detects out-of-order keys

Closed

WT-15292 disagg: test_bug033.py, test_prepare{22,23,27}.py crash in __wt_tree_modify_set called by rollback_to_stable

Closed

WT-15680 Early exit verify in disagg with deltas

Open

WT-15193 Add test case to ensure prefix suffix compression works for page deltas

Closed

WT-15456 test/format (disagg.mode=leader) Transaction rollback assertion fires !WT_IS_METADATA(op->btree->dhandle)

Open

WT-15553 (disagg.mode=leader) test/format internal key OOO

Open

WT-15410 test/format (disagg.mode=leader) Delta chain validation failed

Closed

(4 is related to, 8 related to)

Assignee:: Sean Watt
Reporter:: xgen-buildbaron-user
Collaborators:: Mariam Mojid
Votes:: 0 Vote for this issue
Watchers:: 10 Start watching this issue

Created:: Sep 12 2025 01:51:28 AM UTC
Updated:: Oct 07 2025 05:18:48 AM UTC
Resolved:: Sep 26 2025 01:19:41 AM UTC

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates