Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Transactions - 2025-11-07
Story Points:
8
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-asan-stress
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-test-disagg-leader-1
- format-stress-test-disagg-leader-3
First Failing Revision:
- 73927b2a33767019bef29a9c9f0af0696937f99c(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15826
Count of Linked BFGs (Last 30 days):
2

format-stress-test-disagg-leader-1 on amazon2023-disagg-asan-stress

Host: i-081fbe05243a24d96
Project: wiredtiger
Commit: 73927b2a
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-1

format-stress-test-disagg-leader-1 task_log

Logs:

[1760925333:139088][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_OUT_OF_ORDER][ERROR]: WT_CURSOR.next out-of-order returns: returned key 0000140000.00/opqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij then key 0000140000.00/opqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij
[1760925335:952928][15071:0xffff10c3c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_OUT_OF_ORDER][ERROR]: WT_CURSOR.next out-of-order returns: returned key 0000140000.00/opqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij then key 0000140000.00/opqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij
[1760925339:310043][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310057][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 939: Error at src/cursor/cur_hs.c:939: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310063][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1760925339:310069][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1760925339:310073][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310077][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 939: Error at src/cursor/cur_hs.c:939: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310081][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1760925339:310085][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1760925339:310089][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310093][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 939: Error at src/cursor/cur_hs.c:939: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310097][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1760925339:310101][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1760925339:310105][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310109][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 939: Error at src/cursor/cur_hs.c:939: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310113][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1760925339:310117][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1760925339:310121][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310125][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *), 939: Error at src/cursor/cur_hs.c:939: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310128][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near_helper(WT_SESSION_IMPL *, WT_CURSOR *, _Bool), 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1760925339:310132][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1760925339:310136][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1760925339:310140][15071:0xffff1022c0e0], t, file:T00003.wt_stable, WT_CURSOR.next: [WT_VERB_ERROR_RETURNS][ERROR]: int __curhs_search_near(WT_CURSOR *, int *),

logs

format-stress-test-disagg-leader-1 task_log

Logs:

#0  0x0000ffffac0bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffac0bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffac0763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffac062264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffacbdacf8 [PAC] in __wt_abort (session=0xffffab6b4ac0) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffacea3074 in __wt_panic_func (session=0xffffab6b4ac0, error=22, func=0xffffad04cdc0 <__PRETTY_FUNCTION__.__cursor_key_order_check_row> "int __cursor_key_order_check_row(WT_SESSION_IMPL *, WT_CURSOR_BTREE *, _Bool)", line=637, category=WT_VERB_DEFAULT, fmt=0xffffad04cd20 <str> "found key out-of-order returns") at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/support/err.c:611
#5  0x0000ffffac497f78 in __cursor_key_order_check_row (session=0xffffab6b4ac0, cbt=0x51a0001d8880, next=true) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/btree/bt_curnext.c:637
#6  0x0000ffffac496b3c in __wti_cursor_key_order_check (session=0xffffab6b4ac0, cbt=0x51a0001d8880, next=true) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/btree/bt_curnext.c:658
#7  0x0000ffffac49cc8c in __wt_btcur_next (cbt=0x51a0001d8880, truncating=false) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/btree/bt_curnext.c:1008
#8  0x0000ffffac9255a0 in __curfile_next (cursor=0x51a0001d8880) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/cursor/cur_file.c:185
#9  0x0000ffffac99693c in __clayered_next (cursor=0x516000193880) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/src/cursor/cur_layered.c:837
#10 0x0000aaaae1e0d7a8 in read_op (cursor=0x516000193880, op=NEXT, exactp=0x0) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/test/format/format_inline.h:45
#11 0x0000aaaae1e0ab28 in nextprev (tinfo=0x519000046080, next=true) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/test/format/ops.c:1748
#12 0x0000aaaae1dfd3fc in table_op (tinfo=0x519000046080, intxn=true, iso_level=ISOLATION_SNAPSHOT, op=UPDATE) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/test/format/ops.c:943
#13 0x0000aaaae1df4410 in ops (arg=0x519000046080) at /data/mci/8570f3695c20eeb6763b77888a4dfb7a/wiredtiger/test/format/ops.c:1346
#14 0x0000aaaae1d7de84 in asan_thread_start () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239
#15 0x0000ffffac0bdb78 in start_thread () from /lib64/libc.so.6
#16 0x0000ffffac12acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is related to

WT-15877 Tune SQLite3 to improve performace

Closed

WT-15709 Support generating page deltas for page splits

Backlog

WT-15878 Allow verify to return errors in salvage

Closed

related to

WT-15856 Invariant failure cursor returned out-of-order keys

Open

WT-14700 Discard root pages for old checkpoints

Closed

WT-15748 Add stat to count how many ingest btrees and stable btrees are skipped by eviction server

Closed

WT-15716 (disagg.mode=leader) test/format __reconcile assertion failure

Closed

(2 related to)

Assignee:: Linh Tran
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 5 Start watching this issue

Created:: Oct 22 2025 03:06:35 AM UTC
Updated:: Oct 30 2025 07:33:02 AM UTC
Resolved:: Oct 29 2025 02:48:39 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates