__wt_btree_open, 213: WiredTiger assertion failed on lock_flags while getting checkpoint lock

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-test-disagg-switch-3 on amazon2023-disagg-stress

      Host: i-0f7760914dfc20329
      Project: wiredtiger
      Commit: 0e226ac2
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-3 task_log

      Logs:

      [1772511422:249487][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_btree_open, 213: WiredTiger assertion failed: '!(((session->lock_flags) & (0x0100u)) != 0) || (((session->lock_flags) & (0x0001u)) != 0)'. deadlock
[1772511422:249513][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1772511422:249525][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249531][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249533][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249535][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249537][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249538][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249540][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249542][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249544][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249546][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249548][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249550][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249551][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249553][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249555][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249557][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249559][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249560][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249562][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249565][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249567][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249568][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249570][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249572][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249574][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249576][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249577][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249579][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249581][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249583][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249584][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249586][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249588][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249590][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249592][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249594][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249596][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249598][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curstat_next, 216: Error at src/cursor/cur_stat.c:216: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1772511422:249599][25251:0xffff8b7fdbc0], t, file:T00001.wt_stable/WiredTigerCheckpoint.17, statlog-server:

      logs

      format-stress-test-disagg-switch-3 task_log

      Logs:

      #0  0x0000ffff8c0c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff8c0c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff8c07b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff8c062224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8c437454 [PAC] in __wt_abort (session=session@entry=0x33e13e400fb0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff8c2b0df0 in __wt_btree_open (session=session@entry=0x33e13e400fb0, op_cfg=op_cfg@entry=0x0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/btree/bt_handle.c:213
#5  0x0000ffff8c3550d4 in __wt_conn_dhandle_open (session=session@entry=0x33e13e400fb0, cfg=cfg@entry=0x0, flags=flags@entry=0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/conn/conn_dhandle.c:647
#6  0x0000ffff8c4c4adc in __wt_session_get_dhandle (session=0x33e13e400fb0, uri=0x33e17b1f2cc0 "file:T00001.wt_stable/WiredTigerCheckpoint.17", checkpoint=0x0, cfg=0x0, flags=0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/session/session_dhandle.c:986
#7  0x0000ffff8c4c4fbc in __wt_session_get_dhandle (session=session@entry=0x33e13e400fb0, uri=0x33e17b1f2cc0 "file:T00001.wt_stable/WiredTigerCheckpoint.17", checkpoint=0x0, cfg=cfg@entry=0x0, flags=flags@entry=0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/session/session_dhandle.c:980
#8  0x0000ffff8c3531c0 in __conn_btree_apply_internal (session=session@entry=0x33e13e400fb0, dhandle=dhandle@entry=0x33e13a6ea800, file_func=file_func@entry=0xffff8c36f9f0 <__statlog_apply>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/conn/conn_dhandle.c:744
#9  0x0000ffff8c355674 in __wt_conn_btree_apply (session=session@entry=0x33e13e400fb0, uri=uri@entry=0x0, file_func=file_func@entry=0xffff8c36f9f0 <__statlog_apply>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/conn/conn_dhandle.c:812
#10 0x0000ffff8c36f58c in __statlog_log_one (session=session@entry=0x33e13e400fb0, path=path@entry=0xffff8b7fd2b8, tmp=tmp@entry=0xffff8b7fd290) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/conn/conn_stat.c:460
#11 0x0000ffff8c36f6ec in __statlog_server (arg=0x33e13e400fb0) at /data/mci/8540e7f4580de77bb1577e1c77a279b8/wiredtiger/src/conn/conn_stat.c:545
#12 0x0000ffff8c0c2834 in start_thread () from /lib64/libc.so.6
#13 0x0000ffff8c066e5c [PAC] in thread_start () from /lib64/libc.so.6

      logs

      format-stress-test-disagg-switch-3 task_log

      Logs:

      Adding file to tarball: 'tools/test/binary_files/mongodb_size_mismatch.log'.

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: