test/format (disagg.mode=leader, PALite) Delta chain backlink_lsn mismatch

XMLWordPrintableJSON

      format-stress-test-disagg-leader-palite-2 on amazon2023-disagg-stress

      Host: i-0cf285d13c0dfc33f
      Project: wiredtiger
      Commit: c43f6ff4
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-palite-2 task_log

      Logs:

      [1763685615:660477][6139:0xffffba7fdbc0], t, file:T00001.wt_stable, eviction-server: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:1865: Pages::verify_chain(const std::vector<PageInfo>&, uint64_t)::<lambda(const PageInfo&)>: Delta page base_lsn mismatch: {table_id=41, page_id=1481108, lsn=4920517, backlink_lsn=4720011, base_lsn=4720011, flags=0x2}, full page: {table_id=41, page_id=1481108, lsn=4776699, backlink_lsn=4720011, base_lsn=0, flags=0x0}
[1763685615:660568][6139:0xffffba7fdbc0], t, file:T00001.wt_stable, eviction-server: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:568: int safe_call(WT_SESSION*, S*, MemberFunc, Args&& ...) [with T = PaliteHandle; S = __wt_page_log_handle; MemberFunc = int (PaliteHandle::*)(long unsigned int, long unsigned int, __wt_page_log_put_args*, const __wt_item*); Args = {long unsigned int&, long unsigned int&, __wt_page_log_put_args*&, const __wt_item*&}; WT_SESSION = __wt_session]: Call failed
[1763685615:660588][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660593][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660595][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660597][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660598][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660600][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660602][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660604][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660605][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660607][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660608][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660610][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660611][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660613][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660614][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660616][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 302: Error at src/cursor/cur_hs.c:302: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1763685615:660618][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660619][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660621][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660623][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1763685615:660628][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660631][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660632][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660634][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660635][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1763685615:660637][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660638][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660640][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1763685615:660641][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660643][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660646][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_writelock, 364: Error at src/support/mtx_rw.c:364: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1763685615:660648][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3006: Error at src/evict/evict_lru.c:3006: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1763685615:660649][6139:0xffffba7fdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->

      logs

      format-stress-test-disagg-leader-palite-2 task_log

      Logs:

      #0  0x0000ffffbc8bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffbc8bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffbc8763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffbc862264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffbcc2e254 [PAC] in __wt_abort (session=session@entry=0x3043ffc901e8) at /data/mci/13a3bdcfb390f25ba21ed27bebe47f60/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffbccd2ac4 in __wt_panic_func (session=session@entry=0x3043ffc901e8, error=error@entry=22, func=func@entry=0xffffbcdc31f8 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffbcd63528 "eviction thread error") at /data/mci/13a3bdcfb390f25ba21ed27bebe47f60/wiredtiger/src/support/err.c:611
#5  0x0000ffffbcbe4650 in __evict_thread_run (session=0x3043ffc901e8, thread=0x3043ffe30780) at /data/mci/13a3bdcfb390f25ba21ed27bebe47f60/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffffbcced568 in __thread_run (arg=0x3043ffe30780) at /data/mci/13a3bdcfb390f25ba21ed27bebe47f60/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffffbc8bdb78 in start_thread () from /lib64/libc.so.6
#8  0x0000ffffbc92acdc [PAC] in thread_start () from /lib64/libc.so.6

      logs

      Repro Artifacts:

            Assignee:
            Alex Blekhman
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: