- 
    Type:Build Failure 
- 
    Resolution: Fixed
- 
    Priority:Critical - P2 
- 
    Affects Version/s: None
- 
    Component/s: PALM, Reconciliation
- 
        Storage Engines, Storage Engines - Transactions
- 
        SE Transactions - 2025-09-26, SE Transactions - 2025-10-10
- 
        8
- 
        - format-stress-data-validation-test-disagg-leader-1
- format-stress-data-validation-test-disagg-leader-2
- format-stress-data-validation-test-disagg-leader-3
- format-stress-data-validation-test-disagg-leader-4
- format-stress-data-validation-test-disagg-leader-5
- format-stress-test-disagg-leader-3
- format-stress-test-disagg-leader-4
- format-stress-test-disagg-leader-5
- format-stress-test-disagg-switch-1
- format-stress-test-disagg-switch-2
 
- 
        12
format-stress-test-disagg-leader-5 on amazon2023-disagg-stress
Host: i-0ef701305c03cfcc0
Project: wiredtiger
Commit: bd06c967
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
Logs:
[1757384931:112393][5366:0xffff1f2adb80], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid argument: palm_handle_verify_page:723: Delta chain validation failed at position 2: matches.base_lsn != last_base_lsn. Page details: table_id=57, page_id=1521, lsn=589766, flags=10002, matches.base_lsn=584761, last_base_lsn=584765 [1757384931:112418][5366:0xffff1f2adb80], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid error code: /data/mci/64c003ddef0fc6483354ea6795dff899/wiredtiger/ext/page_log/palm/palm.c: 790: "palm_handle_verify_page(plh, session, page_id, lsn)": failed
Logs:
#0 0x0000ffffab6bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffab6bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffab6763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffab662264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffffab66f9a8 [PAC] in __assert_fail_base () from /lib64/libc.so.6 #4 0x0000ffffab66fa04 [PAC] in __assert_fail () from /lib64/libc.so.6 #5 0x0000ffffacc9a250 [PAC] in ?? () #6 0x5f657361622e7365 in ?? ()
Repro Artifacts:
- is depended on by
- 
                    WT-15524 PALM verification needs to consider failed reconciliation writes -         
- Closed
 
-         
- is duplicated by
- 
                    WT-15319 palm_handle_verify_page: Delta chain validation failed -         
- Closed
 
-         
- is related to
- 
                    WT-15441 Fix __clayered_compare failure -         
- Closed
 
-         
- 
                    WT-15528 Ensure read-only page is loaded as clean page -         
- Closed
 
-         
- 
                    WT-15531 Fix error log handling in test_layered31 -         
- Closed
 
-         
- 
                    WT-15575 Prevent the follower from writing checkpoint metadata -         
- Closed
 
-         
- 
                    WT-14996 Set base and backlink LSNs for page discard (tombstone write) -         
- Closed
 
-         
- 
                    WT-15514 Disagg python testing: skip test_import*.py tests -         
- Closed
 
-         
- 
                    WT-15535 Ensure checkpoint cleanup doesn't mark any readonly btree dirty -         
- Closed
 
-         
- 
                    WT-15563 Investigate making cache tolerant to change app step-wise eviction to incremental eviction -         
- Closed
 
-         
- 
                    WT-15568 Add the ability to dump the error log without the connection object -         
- Closed
 
-         
- 
                    WT-15426 Keep a log of errors to pinpoint the origin of the error -         
- Closed
 
-         
- 
                    WT-15541 Improve error pinpoint -         
- Closed
 
-         
- 
                    WT-15576 Use EIO to indicate I/O errors in disagg -         
- Closed
 
-         
- 
                    WT-15455 Don't skip prepared update pages during cursor walk -         
- Closed
 
-         
- 
                    WT-15319 palm_handle_verify_page: Delta chain validation failed -         
- Closed
 
-         
- 
                    WT-15450 test/format (disagg.mode=leader) verify failure unpacking addr -         
- Closed
 
-         
- 
                     WT-15471
        Fix error handling for lsn validation in palm_handle_discard WT-15471
        Fix error handling for lsn validation in palm_handle_discard-         
- Closed
 
-         
- 
                    WT-15550 layered38, 32 data mismatch error -         
- Closed
 
-         
- 
                     WT-15566
        Remove table_id parameter from palm_handle_get_page_ids WT-15566
        Remove table_id parameter from palm_handle_get_page_ids-         
- Closed
 
-         
- related to
- 
                    WT-15292 disagg: test_bug033.py, test_prepare{22,23,27}.py crash in __wt_tree_modify_set called by rollback_to_stable -         
- Closed
 
-         
- 
                    WT-15571 Fix `configure-combinations` build -         
- Closed
 
-         
- 
                    WT-15103 Enable disagg python testing on Evergreen waterfall -         
- Closed
 
-         
- 
                    WT-15193 Add test case to ensure prefix suffix compression works for page deltas -         
- Closed
 
-         
- 
                    WT-15563 Investigate making cache tolerant to change app step-wise eviction to incremental eviction -         
- Closed
 
-         
- 
                    WT-15455 Don't skip prepared update pages during cursor walk -         
- Closed
 
-         
- 
                    WT-15450 test/format (disagg.mode=leader) verify failure unpacking addr -         
- Closed
 
-         
- 
                    WT-15550 layered38, 32 data mismatch error -         
- Closed
 
-         
- 
                     WT-15566
        Remove table_id parameter from palm_handle_get_page_ids WT-15566
        Remove table_id parameter from palm_handle_get_page_ids-         
- Closed
 
-