Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Critical - P2
Fix Version/s: None
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Foundations
Sprint:
SE Foundations - Q4+ Backlog
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-test-disagg-leader-2
- format-stress-test-disagg-leader-3
First Failing Revision:
- ec2c91c5217e4886634c21a04d62fd5e273637e5(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16092
Count of Linked BFGs (Last 30 days):
3

format-stress-test-disagg-leader-2 on amazon2023-disagg-stress

Host: i-048bb04ca10e4b34c
Project: wiredtiger
Commit: de1746fa
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-2

format-stress-test-disagg-leader-2 task_log

Logs:

[1763694500:114253][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_OUT_OF_ORDER][ERROR]: WT_CURSOR.prev out-of-order returns: returned key 0000055055.00/opqrstuvwxyzab then key 0000054083.00/opqrs
[1763694506:003376][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003386][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003388][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003391][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003393][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003395][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003396][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003398][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003399][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003401][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003402][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003404][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003405][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003407][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003409][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003410][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003411][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003413][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003414][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003416][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003417][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003419][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003420][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003423][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003425][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003426][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003427][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003429][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003430][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003434][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003436][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003437][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003438][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003440][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003442][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003443][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003444][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003446][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003447][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1763694506:003449][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1763694506:003450][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
[1763694506:003452][10536:0xffff17fccbc0], t, file:T00001.wt_stable,

logs

format-stress-test-disagg-leader-2 task_log

Logs:

#0  0x0000ffffa6cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffa6cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffa6c763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffa6c62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffa702e254 [PAC] in __wt_abort (session=session@entry=0x708e7fc99de0) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffa70d2ad4 in __wt_panic_func (session=session@entry=0x708e7fc99de0, error=error@entry=22, func=func@entry=0xffffa71b1e88 <__PRETTY_FUNCTION__.46> "__cursor_key_order_check_row", line=line@entry=639, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffa711fad0 "found key out-of-order returns") at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/support/err.c:611
#5  0x0000ffffa6e78be4 in __cursor_key_order_check_row (session=0x708e7fc99de0, cbt=<optimized out>, next=<optimized out>) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curnext.c:639
#6  0x0000ffffa6e80590 in __wti_cursor_key_order_check (session=session@entry=0x708e7fc99de0, cbt=cbt@entry=0x708e7adf8f00, next=next@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curnext.c:660
#7  0x0000ffffa6e8ae44 in __wt_btcur_prev (cbt=cbt@entry=0x708e7adf8f00, truncating=truncating@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curprev.c:950
#8  0x0000ffffa6f87630 in __curfile_prev (cursor=0x708e7adf8f00) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_file.c:245
#9  0x0000ffffa6fa3560 in __layered_prev_int (session=session@entry=0x708e7fc99de0, cursor=cursor@entry=0x708e70db2b40) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_layered.c:911
#10 0x0000ffffa6fa3a0c in __layered_prev (cursor=0x708e70db2b40) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_layered.c:943
#11 0x0000000000412720 in read_op (cursor=cursor@entry=0x708e70db2b40, op=op@entry=PREV, exactp=exactp@entry=0x0) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/format_inline.h:48
#12 0x0000000000413458 in nextprev (tinfo=tinfo@entry=0x708e7f3ffc00, next=next@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:1753
#13 0x00000000004150cc in table_op (tinfo=tinfo@entry=0x708e7f3ffc00, intxn=intxn@entry=true, iso_level=iso_level@entry=ISOLATION_SNAPSHOT, op=<optimized out>, op@entry=UPDATE) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:948
#14 0x0000000000416688 in ops (arg=0x708e7f3ffc00) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:1351
#15 0x0000ffffa6cbdb78 in start_thread () from /lib64/libc.so.6
#16 0x0000ffffa6d2acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is caused by

WT-15618 Implement rewriting internal page deltas directly into a full page

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Nov 23 2025 09:50:57 PM UTC
Updated:: Nov 24 2025 11:04:27 PM UTC
Resolved:: Nov 24 2025 10:55:05 PM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates