Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Disagg CI-blocker, Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines, Storage Engines - Persistence
Sprint:
SE Persistence - 2025-11-21
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-asan-stress
- amazon2023-disagg-stress
Failing Tasks:
First Failing Revision:
- c80600e99d70cbffc50c4673585e431063da3dda(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15988
Count of Linked BFGs (Last 30 days):
8

format-stress-test-disagg-leader-1 on amazon2023-disagg-asan-stress

Host: i-0493c9303b0bebd47
Project: wiredtiger
Commit: 397df101
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-1

format-stress-test-disagg-leader-1 task_log

Logs:

[1762218669:524585][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid argument (22): palm_handle_verify_page:922: Delta chain validation failed at position 1: matches.base_lsn != matched_pages[0].lsn. Page details: table_id=73, page_id=7865, lsn=18165, flags=0x10002, matches.base_lsn=7888, matched_pages[0].lsn=18116
[1762218669:524621][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid error code (-31802): /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/ext/page_log/palm/palm.c:1011: "palm_handle_verify_page(plh, session, page_id, lsn)": failed
[1762218669:524634][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_try_writelock(WT_SESSION_IMPL *, WT_RWLOCK *), 363: Error at src/support/mtx_rw.c:363: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1762218669:524645][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __evict_page(WT_SESSION_IMPL *, _Bool), 2979: Error at src/evict/evict_lru.c:2979: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1762218669:524650][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_try_readlock(WT_SESSION_IMPL *, WT_RWLOCK *), 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1762218669:524655][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __evict_reconcile(WT_SESSION_IMPL *, WT_REF *, uint32_t), 1082: Error at src/evict/evict_page.c:1082: "ret" failed: Device or resource busy
[1762218669:524660][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_evict(WT_SESSION_IMPL *, WT_REF *, WT_REF_STATE, uint32_t), 241: Error at src/evict/evict_page.c:241: "__evict_reconcile(session, ref, flags)" failed: Device or resource busy
[1762218669:524665][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __evict_reconcile(WT_SESSION_IMPL *, WT_REF *, uint32_t), 1082: Error at src/evict/evict_page.c:1082: "ret" failed: Device or resource busy
[1762218669:524669][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_evict(WT_SESSION_IMPL *, WT_REF *, WT_REF_STATE, uint32_t), 241: Error at src/evict/evict_page.c:241: "__evict_reconcile(session, ref, flags)" failed: Device or resource busy
[1762218669:524674][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __evict_reconcile(WT_SESSION_IMPL *, WT_REF *, uint32_t), 1082: Error at src/evict/evict_page.c:1082: "ret" failed: Device or resource busy
[1762218669:524678][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_evict(WT_SESSION_IMPL *, WT_REF *, WT_REF_STATE, uint32_t), 241: Error at src/evict/evict_page.c:241: "__evict_reconcile(session, ref, flags)" failed: Device or resource busy
[1762218669:524683][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __evict_page(WT_SESSION_IMPL *, _Bool), 2979: Error at src/evict/evict_lru.c:2979: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1762218669:524688][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_ref_block_free(WT_SESSION_IMPL *, WT_REF *, _Bool), 1723: Error at src/include/btree_inline.h:1723: "__wt_btree_block_free(session, addr.addr, addr.size)" failed: WT_ERROR: non-specific WiredTiger error
[1762218669:524693][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __rec_write_wrapup(WT_SESSION_IMPL *, WTI_RECONCILE *), 3093: Error at src/reconcile/rec_write.c:3093: "__wt_ref_block_free(session, ref, r->multi_next == 1)" failed: WT_ERROR: non-specific WiredTiger error
[1762218669:524698][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, _Bool *), 387: Error at src/reconcile/rec_write.c:387: "__rec_write_wrapup(session, r)" failed: WT_ERROR: non-specific WiredTiger error
[1762218669:524702][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, _Bool *), 442: reconciliation failed after building the disk image: WT_ERROR: non-specific WiredTiger error
[1762218669:524706][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, _Bool *), 442: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1762218669:524733][11663:0xffff136a30e0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 29: aborting WiredTiger library

logs

format-stress-test-disagg-leader-1 task_log

Logs:

#0  0x0000ffff9bcbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff9bcbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff9bc763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff9bc62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff9c7e4e50 [PAC] in __wt_abort (session=0xffff9afac9e8) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff9caaf018 in __wt_panic_func (session=0xffff9afac9e8, error=-31802, func=0xffff9cdbde00 <__PRETTY_FUNCTION__.__reconcile> "int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, _Bool *)", line=442, category=WT_VERB_DEFAULT, fmt=0xffff9cdbe460 <str> "reconciliation failed after building the disk image") at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/support/err.c:611
#5  0x0000ffff9c91ae34 in __reconcile (session=0xffff9afac9e8, ref=0x50e0001ff2e0, salvage=0x0, flags=4736, page_lockedp=0xffff164bc020) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/reconcile/rec_write.c:442
#6  0x0000ffff9c914b34 in __wt_reconcile (session=0xffff9afac9e8, ref=0x50e0001ff2e0, salvage=0x0, flags=4736) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/reconcile/rec_write.c:128
#7  0x0000ffff9c6d5930 in __evict_reconcile (session=0xffff9afac9e8, ref=0x50e0001ff2e0, evict_flags=0) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_page.c:1069
#8  0x0000ffff9c6d1998 in __wt_evict (session=0xffff9afac9e8, ref=0x50e0001ff2e0, previous_state=3 '\003', flags=0) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_page.c:241
#9  0x0000ffff9c6aa5ac in __evict_page (session=0xffff9afac9e8, is_server=true) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_lru.c:3013
#10 0x0000ffff9c6af988 in __evict_lru_pages (session=0xffff9afac9e8, is_server=true) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_lru.c:1412
#11 0x0000ffff9c6b0520 in __evict_pass (session=0xffff9afac9e8) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_lru.c:918
#12 0x0000ffff9c6aeb98 in __evict_server (session=0xffff9afac9e8, did_work=0xffff15db4ca0) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_lru.c:463
#13 0x0000ffff9c6a1d2c in __evict_thread_run (session=0xffff9afac9e8, thread=0x507000003d60) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/evict/evict_lru.c:331
#14 0x0000ffff9cb60d4c in __thread_run (arg=0x507000003d60) at /data/mci/4533b6400ccb387c1dbf26412c402fdf/wiredtiger/src/support/thread_group.c:32
#15 0x0000ffff9d3057a0 in asan_thread_start () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239
#16 0x0000ffff9bcbdb78 in start_thread () from /lib64/libc.so.6
#17 0x0000ffff9bd2acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

wt_15988.patch
4 kB
Nov 13 2025 05:55:25 AM UTC
stderr.palm.2.log
5 kB
Nov 13 2025 05:56:07 AM UTC
stderr.palm.1.log
5 kB
Nov 13 2025 05:56:07 AM UTC
stderr.palite.log
29 kB
Nov 13 2025 05:56:06 AM UTC
image-2025-11-13-16-15-40-613.png
593 kB
Nov 13 2025 05:15:42 AM UTC

duplicates

WT-16017 Force to use a new page id if we failed to write a replacement page

Closed

is related to

WT-16017 Force to use a new page id if we failed to write a replacement page

Closed

related to

WT-15990 test/format (disagg.mode=leader, PALite) failed with Device or resource busy

Open

Assignee:: Albert Song
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 4 Start watching this issue

Created:: Nov 08 2025 04:17:35 AM UTC
Updated:: Nov 14 2025 03:07:15 AM UTC
Resolved:: Nov 14 2025 03:07:16 AM UTC

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates