Loading...

Type: Build Failure
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: PALM
Labels:
- BB-Tools

Assigned Teams:

Storage Engines
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-data-validation-test-disagg-leader-3
First Failing Revision:
- ce2200f38c996c74e26198c96a879c13dc6a9aa7(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15909
Count of Linked BFGs (Last 30 days):
1

format-stress-data-validation-test-disagg-leader-3 on amazon2023-disagg-stress

Host: i-0e8b2d1105c6cfa19
Project: wiredtiger
Commit: ce2200f3
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-data-validation-test-disagg-leader-3

format-stress-data-validation-test-disagg-leader-3 task_log

Logs:

[1761612941:025250][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid argument (22): palm_handle_verify_page:922: Delta chain validation failed at position 1: matches.base_lsn != matched_pages[0].lsn. Page details: table_id=49, page_id=20152, lsn=85508, flags=0x10002, matches.base_lsn=28578, matched_pages[0].lsn=59985
[1761612941:025270][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_EXTENSION][ERROR]: palm LMDB: Invalid error code (-31802): /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/ext/page_log/palm/palm.c:1010: "palm_handle_verify_page(plh, session, page_id, lsn)": failed
[1761612941:025277][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1761612941:025283][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1761612941:025285][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1761612941:025287][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1761612941:025289][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1761612941:025290][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1761612941:025292][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
[1761612941:025293][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
[1761612941:025295][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_ref_block_free, 1723: Error at src/include/btree_inline.h:1723: "__wt_btree_block_free(session, addr.addr, addr.size)" failed: WT_ERROR: non-specific WiredTiger error
[1761612941:025297][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write_wrapup, 3086: Error at src/reconcile/rec_write.c:3086: "__wt_ref_block_free(session, ref, r->multi_next == 1)" failed: WT_ERROR: non-specific WiredTiger error
[1761612941:025299][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_ERROR_RETURNS][ERROR]: __reconcile, 387: Error at src/reconcile/rec_write.c:387: "__rec_write_wrapup(session, r)" failed: WT_ERROR: non-specific WiredTiger error
[1761612941:025301][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __reconcile, 442: reconciliation failed after building the disk image: WT_ERROR: non-specific WiredTiger error
[1761612941:025302][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __reconcile, 442: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1761612941:025305][10402:0xffff1df6dbc0], t, file:T00002.wt_stable, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

logs

format-stress-data-validation-test-disagg-leader-3 task_log

Logs:

#0  0x0000ffffb80bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffb80bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffb80763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffb8062264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffb8427448 [PAC] in __wt_abort (session=session@entry=0x5287bfca2580) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffb84cb20c in __wt_panic_func (session=session@entry=0x5287bfca2580, error=error@entry=-31802, func=func@entry=0xffffb85bf010 <__PRETTY_FUNCTION__.63> "__reconcile", line=line@entry=442, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffb8579b40 "reconciliation failed after building the disk image") at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/support/err.c:611
#5  0x0000ffffb8473dac in __reconcile (session=session@entry=0x5287bfca2580, ref=ref@entry=0x5287bfff5400, salvage=salvage@entry=0x0, flags=flags@entry=516, page_lockedp=page_lockedp@entry=0xffff1df6bdbf) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/reconcile/rec_write.c:442
#6  0x0000ffffb8472834 in __wt_reconcile (session=session@entry=0x5287bfca2580, ref=0x5287bfff5400, salvage=salvage@entry=0x0, flags=flags@entry=516) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/reconcile/rec_write.c:128
#7  0x0000ffffb82ea8d4 in __wt_sync_file (session=session@entry=0x5287bfca2580, syncop=syncop@entry=WT_SYNC_CHECKPOINT) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/btree/bt_sync.c:364
#8  0x0000ffffb8325b40 in __checkpoint_tree (session=session@entry=0x5287bfca2580, is_checkpoint=is_checkpoint@entry=true, cfg=cfg@entry=0xffff1df6ce18) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:2576
#9  0x0000ffffb83261f8 in __checkpoint_tree_helper (session=0x5287bfca2580, cfg=0xffff1df6ce18) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:2714
#10 0x0000ffffb831ec44 in __checkpoint_apply_to_dhandles (session=session@entry=0x5287bfca2580, cfg=cfg@entry=0xffff1df6ce18, op=op@entry=0xffffb8326174 <__checkpoint_tree_helper>) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:340
#11 0x0000ffffb8326e30 in __checkpoint_db_internal (session=session@entry=0x5287bfca2580, cfg=cfg@entry=0xffff1df6ce18) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:1356
#12 0x0000ffffb832820c in __checkpoint_db_wrapper (session=session@entry=0x5287bfca2580, cfg=cfg@entry=0xffff1df6ce18) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:1750
#13 0x0000ffffb83285ec in __wt_checkpoint_db (session=session@entry=0x5287bfca2580, cfg=cfg@entry=0xffff1df6ce18, waiting=waiting@entry=true) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/checkpoint/checkpoint_txn.c:1829
#14 0x0000ffffb84a9d68 in __session_checkpoint (wt_session=0x5287bfca2580, config=<optimized out>) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/src/session/session_api.c:2439
#15 0x0000000000405dd0 in checkpoint (arg=<optimized out>) at /data/mci/63966fbff14a4e9d3864c28a42de42ac/wiredtiger/test/format/checkpoint.c:147
#16 0x0000ffffb80bdb78 in start_thread () from /lib64/libc.so.6
#17 0x0000ffffb812acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

duplicates

WT-15681 PALM validation fails when there is a failed write

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates