Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Fixed
Priority: Critical - P2
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: PALM, Reconciliation
Labels:
- BB-Tools
- CI-blocker
- Disag_Storage
- dc

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Transactions - 2025-09-26, SE Transactions - 2025-10-10
Story Points:
8
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
First Failing Revision:
- bd06c96780ac22a61fb1d5bc43d416c67c93f05a(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15410
Count of Linked BFGs (Last 30 days):
15

format-stress-test-disagg-leader-5 on amazon2023-disagg-stress

Host: i-0ef701305c03cfcc0
Project: wiredtiger
Commit: bd06c967
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-leader-5

format-stress-test-disagg-leader-5 task_log

Logs:

[1757384931:112393][5366:0xffff1f2adb80], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid argument: palm_handle_verify_page:723: Delta chain validation failed at position 2: matches.base_lsn != last_base_lsn. Page details: table_id=57, page_id=1521, lsn=589766, flags=10002, matches.base_lsn=584761, last_base_lsn=584765
[1757384931:112418][5366:0xffff1f2adb80], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid error code: /data/mci/64c003ddef0fc6483354ea6795dff899/wiredtiger/ext/page_log/palm/palm.c: 790: "palm_handle_verify_page(plh, session, page_id, lsn)": failed

logs

format-stress-test-disagg-leader-5 task_log

Logs:

#0  0x0000ffffab6bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffab6bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffab6763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffab662264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffab66f9a8 [PAC] in __assert_fail_base () from /lib64/libc.so.6
#4  0x0000ffffab66fa04 [PAC] in __assert_fail () from /lib64/libc.so.6
#5  0x0000ffffacc9a250 [PAC] in ?? ()
#6  0x5f657361622e7365 in ?? ()

logs

Repro Artifacts:

CONFIG

is depended on by

WT-15524 PALM verification needs to consider failed reconciliation writes

Closed

is duplicated by

WT-15319 palm_handle_verify_page: Delta chain validation failed

Closed

is related to

WT-15441 Fix __clayered_compare failure

Closed

WT-15528 Ensure read-only page is loaded as clean page

Closed

WT-15531 Fix error log handling in test_layered31

Closed

WT-15575 Prevent the follower from writing checkpoint metadata

Closed

WT-14996 Set base and backlink LSNs for page discard (tombstone write)

Closed

WT-15514 Disagg python testing: skip test_import*.py tests

Closed

WT-15535 Ensure checkpoint cleanup doesn't mark any readonly btree dirty

Closed

WT-15563 Investigate making cache tolerant to change app step-wise eviction to incremental eviction

Closed

WT-15568 Add the ability to dump the error log without the connection object

Closed

WT-15426 Keep a log of errors to pinpoint the origin of the error

Closed

WT-15541 Improve error pinpoint

Closed

WT-15576 Use EIO to indicate I/O errors in disagg

Closed

WT-15455 Don't skip prepared update pages during cursor walk

Closed

WT-15319 palm_handle_verify_page: Delta chain validation failed

Closed

WT-15450 test/format (disagg.mode=leader) verify failure unpacking addr

Closed

WT-15471 Fix error handling for lsn validation in palm_handle_discard

Closed

WT-15550 layered38, 32 data mismatch error

Closed

WT-15566 Remove table_id parameter from palm_handle_get_page_ids

Closed

related to

WT-15292 disagg: test_bug033.py, test_prepare{22,23,27}.py crash in __wt_tree_modify_set called by rollback_to_stable

Closed

WT-15571 Fix `configure-combinations` build

Closed

WT-15103 Enable disagg python testing on Evergreen waterfall

Closed

WT-15193 Add test case to ensure prefix suffix compression works for page deltas

Closed

WT-15563 Investigate making cache tolerant to change app step-wise eviction to incremental eviction

Closed

WT-15455 Don't skip prepared update pages during cursor walk

Closed

WT-15450 test/format (disagg.mode=leader) verify failure unpacking addr

Closed

WT-15550 layered38, 32 data mismatch error

Closed

WT-15566 Remove table_id parameter from palm_handle_get_page_ids

Closed

(15 is related to, 9 related to)

Assignee:: Chenhao Qu
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 11 Start watching this issue

Created:: Sep 09 2025 03:02:07 AM UTC
Updated:: Sep 29 2025 12:53:00 AM UTC
Resolved:: Sep 26 2025 06:52:39 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates